System for in-line inspection of seal integrity

ABSTRACT

A process for monitoring seal quality of a packaging article. In one embodiment, the process for monitoring the seal quality includes sealing a film to form packaging article and forming a seal area. The seal area is analyzed by a vision system to acquire image data of the seal area. A vision inspection engine analyzes the image date to determine the continuity of the seal, the strength of the seal, or both.

BACKGROUND

The subject matter disclosed herein relates to a system and process forinspecting packaging products for quality assurance, to ensure that sealintegrity of the packaging product.

Flexible films are frequently used in packaging because they are capableof conforming to surfaces of products. The quality of packaging dependson the quality of the film material and the effective closure of anyseals. Adequate seal quality implies the complete fusion of two opposingseal layers. In heat sealing, this fusion is achieved by applying aspecific combination of temperature and pressure for a certain period oftime, allowing the long chain molecules of the seal layers to join. Sealdefects can compromise the seal by an inadequate combination of thesealing parameters or by the presence of water vapor, air bubbles,wrinkles, or product contamination in the seal. Such seal defects can beburnt seals, cold seals, weak seals, pleats, holes, contamination in theseal area, seal burns, seal voids and localized defects. Seal defectscan allow air and microorganisms to penetrate the package and spoilenclosed food. They can also negatively impact seal appearance, causingloss of consumer confidence in packaging integrity.

Detection of faulty seals remains today as a labor-intensive, offline,destructive process. Current seal test methods are described in ASTMF1921 “Standard Test Methods for Hot Seal Strength (Hot Tack) ofThermoplastic Polymers and Blends Comprising the Sealing Surfaces ofFlexible Webs,” ASTM F88 “Standard Test Method for Seal Strength ofFlexible Barrier Materials,” ASTM F2054 “Standard Test Method for BurstTesting of Flexible Package Seals Using Internal Air PressurizationWithin Restraining Plates,” and ASTM F2095 “Standard Test Methods forPressure Decay Leak Test for Flexible Packages With and WithoutRestraining Plates.” Traditional test methods are destructive and canonly be performed offline. In addition, these processes only allow thetesting of a limited number of packaging articles (such as less than 1%)and result in the destruction of the packaging article.

It would be desirable to be able to check packaging product seals in anon-destructive process. It would further be desirable to be able checkseals in-line with the manufacture of the packaging product or withpackaging of an article within the packaging product prior to pack offof the packaging product.

The discussion above is merely provided for general backgroundinformation and is not intended to be used as an aid in determining thescope of the claimed subject matter.

BRIEF SUMMARY

A process for monitoring seal quality of a packaging article. In oneembodiment, the process for monitoring the seal quality includes sealinga film to form packaging article and forming a seal area. The seal areais analyzed by a vision system to acquire image data of the seal area. Avision inspection engine analyzes the image data to determine thecontinuity of the seal, the strength of the seal, or both.

In one exemplary embodiment, a process for monitoring seal quality of apackaging article is disclosed. The process for monitoring seal qualityof a packaging article comprises

-   -   A) sealing a film to itself, another film, or a packaging        support to form a packaging article by forming at least one seal        area;    -   B) acquiring image data of the at least one seal area with a        vision system comprising an image capture device;    -   C) assessing the image data of the seal area with a vision        inspection engine to verify the continuity of the seal, the        strength of the seal, or both the continuity and strength of the        seal area.

In an exemplary embodiment the forming of the at least one seal area isformed by a heat generated seal.

In an exemplary embodiment the heat generated seal is a non-linear seal.

In an exemplary embodiment the heat generated seal is selected from thegroup consisting of an impulse seal, an ultrasonic seal, a laser sealand a heat seal.

In an exemplary embodiment the film comprises at least one layercontaining a fluorescence-based indicator.

In an exemplary embodiment the vision system further comprising a blueband pass filter.

In an exemplary embodiment the the vision system is an ultravioletvision system further comprising an ultraviolet light source.

In an exemplary embodiment the ultraviolet vision system furthercomprises a white light source.

In an exemplary embodiment the ultraviolet vision system illuminates thewhite light source and the ultraviolet light source in a variablepattern.

In an exemplary embodiment the image capture device is capable ofcapturing visual images in the visual spectrum.

In an exemplary embodiment the process for monitoring seal quality of apackaging article further comprises the steps of:

-   -   A) exposing the packaging article to incident radiation to        excite the fluorescence-based indicator so that the        fluorescence-based indicator fluoresces;    -   B) acquiring image data of the fluorescence emitted from the        seal area by the packaging article, while the indicator is        fluorescing.

In an exemplary embodiment the fluorescence-based indicator comprises atleast one member selected from the group consisting ofultraviolet-indicator, infrared-indicator, dye, pigment, opticalbrightener, fluorescent whitening agent,2,2′-(2,5-thiophenylenediyl)bis(5-tert-butylbenzoxazole),hydroxyl-4-)p-tolylamino)anthracene-9,10-dione,2,5-thiophenediylbis(5-tert-butyl-1,3-benzoxazole), and anthraquinonedyestuff.

In an exemplary embodiment the fluorescence-based indicator is presentin at least one layer of the film, with the fluorescence-based indicatorbeing present at a level of from 0.5 to 1000 ppm, based on layer weight.

In an exemplary embodiment the fluorescence-based indicator is presentin at least one layer of the film, with the fluorescence-based indicatorbeing present at a level of from 5 to 10 ppm, based on layer weight.

In an exemplary embodiment the vision inspection engine comprises acomputing apparatus comprising computer executable instructionsconfigured to determine whether fluorescent electromagnetic energyemitted by the excited fluorescence-based indicator is indicative of adefective seal.

In an exemplary embodiment the computer executable instructions comprisean artificial intelligent algorithm.

In an exemplary embodiment the artificial intelligent algorithm with atraining set of training packaging articles before analyzing the imagedata.

In an exemplary embodiment determining that the seal is defectivecomprises determining that the film exhibits a higher or lower intensityof fluorescence in a first region of the seal, relative to a secondregion of the seal.

In an exemplary embodiment determining that the seal is defectivecomprises determining that the film exhibits a higher or lower intensityof fluorescence in a first region of the seal, relative to an expectedlevel fluorescence.

In an exemplary embodiment determining that the seal is defectivecomprises determining at least one of (i) that the film exhibits ahigher or lower intensity of fluorescence in a first region of the seal,relative to a second region of the seal, (ii) that the film exhibits ahigher or lower intensity of fluorescence in a first region of the seal,relative to an expected level fluorescence, or (iii) both (i) and (ii).

In an exemplary embodiment the image data is time-delayed thermographydata captured at a time after the forming of at least one seal area.

In an exemplary embodiment the vision system is a thermography visionsystem comprising an infrared imaging device capable of capturing atemperature distribution based on the amount of infrared radiationemitted from the seal area.

In an exemplary embodiment the image data is taken between 2 and 30seconds after the forming at least one seal area.

In an exemplary embodiment the image data is taken between 5 and 20seconds after the forming at least one seal area.

In an exemplary embodiment the process further comprises an activecooling system.

In an exemplary embodiment the vision inspection engine comprises acomputing apparatus comprising computer executable instructionsconfigured to determine whether thermal image data captured of the sealis indicative of a defective seal.

In an exemplary embodiment the computer executable instructions comprisean artificial intelligent algorithm.

In an exemplary embodiment the process further comprises generating theartificial intelligent algorithm with a training set of trainingpackaging articles before analyzing the image data.

In an exemplary embodiment determining that the seal is defectivecomprises determining that the film exhibits a higher or lower thermaltemperature in a first region of the seal, relative to a second regionof the seal.

In an exemplary embodiment the vision system is a photoelasticity visionsystem comprising:

-   -   (i) a first linear polarizer having a direction of polarization        oriented in a first direction;    -   (ii) a second linear polarizer have a direction of polarization        oriented orthogonal to the first direction;    -   (iii) a light source; and    -   (iv) an imaging device.

In an exemplary embodiment the first linear polarizer is integrated intothe light source.

In an exemplary embodiment the second linear polarizer is integratedinto the imaging device.

In an exemplary embodiment the light source is a white light sourcehaving a wavelength spectrum from 400 nm to 700 nm.

In an exemplary embodiment the white light source is a diffused whitelight source.

In an exemplary embodiment the process further comprises a lightdiffuser positioned between the light source and the packaging article.

In an exemplary embodiment the acquiring of photoelasticity image datais performed in-line with the forming of the packaging article.

In an exemplary embodiment the vision system is a first vision systemand further comprising second vision system distinct from the firstvision system.

In an exemplary embodiment the first vision system is a thermographyvision system comprising an infrared imaging device; and the secondvision system is a photoelasticity vision system comprising:

-   -   (i) a first linear polarizer having a direction of polarization        oriented in a first direction;    -   (ii) a second linear polarizer have a direction of polarization        oriented orthogonal to the first direction;    -   (iii) a light source; and    -   (iv) an image capture device.

In an exemplary embodiment the process further comprises a third visionsystem, the third vision system being an ultraviolet vision systemcomprising an ultraviolet light source.

In an exemplary embodiment the first vision system is a thermographyvision system comprising an infrared imaging device; and the secondvision system is an ultraviolet vision system comprising an ultravioletlight source.

In an exemplary embodiment the process further comprises a third visionsystem, the third vision system is a photoelasticity vision systemcomprising:

-   -   (i) a first linear polarizer having a direction of polarization        oriented in a first direction;    -   (ii) a second linear polarizer have a direction of polarization        oriented orthogonal to the first direction;    -   (iii) a light source; and    -   (iv) an image capture device.

In an exemplary embodiment the first vision system is an ultravioletvision system comprising an ultraviolet light source; and the secondvision system is a photoelasticity vision system comprising:

-   -   (i) a first linear polarizer having a direction of polarization        oriented in a first direction;    -   (ii) a second linear polarizer have a direction of polarization        oriented orthogonal to the first direction;    -   (iii) a light source; and    -   (iv) an image capture device.

In an exemplary embodiment the process further comprises a third visionsystem, the third vision system is a thermography vision systemcomprising an infrared imaging device.

In an exemplary embodiment the vision inspection engine comprises acomputing apparatus comprising computer executable instructionsconfigured to determine whether visual image data captured of the sealis indicative of a defective seal.

In an exemplary embodiment the film has a total free shrink in eitherthe machine or traverse direction at 85° C., of less than 10 percent.

In an exemplary embodiment the film has a total free shrink in eitherthe machine or traverse direction at 85° C., of at least 10 percent.

In an exemplary embodiment determining that the seal is defective isindicative of at least one of a gap in a seal of the packaging article,a pleat in the seal of the packaging article, a weak seal or a coldseal.

In an exemplary embodiment the product is a food product.

In an exemplary embodiment the food product comprises at least onemember selected from the group consisting of meat and cheese.

In an exemplary embodiment the film is a monolayer film.

In an exemplary embodiment the film is a multilayer film.

In an exemplary embodiment the multilayer film comprises a functionallayer; and wherein the fluorescence-based indicator is present in thefunctional layer.

In an exemplary embodiment the functional layer is a member selectedfrom the group consisting of oxygen barrier layer, organoleptic barrierlayer, moisture barrier layer, hazardous chemical barrier layer,microbial barrier layer, acid layer, acid salt layer, bacteriocin layer,bacteriophage layer, metal layer, metal salt layer, natural oil layer,natural extract layer, layer containing polyhexamethylene biguanidehydrochloride, layer containing paraben, layer containing graftedsilane-quaternary amine, layer containing triclosan, layer containingzeolite of silver, copper, and/or zinc.

In an exemplary embodiment the functional layer is an oxygen barrierlayer comprising at least one member selected from the group consistingof vinylidene chloride copolymer, saponified ethylene/vinyl acetatecopolymer, polyamide, polyester, oriented polypropylene, and ethylenehomopolymer.

In an exemplary embodiment the multilayer film comprises:

-   -   A) a first layer which is a first outer film layer configured to        serve as a heat seal layer;    -   B) a second layer which is a second outer layer configured to        serve as an abuse layer;    -   C) a third layer which is between the first layer and the second        layer, wherein the third layer is configured to serve as a        functional layer;    -   D) a fourth layer which is between the first layer and the third        layer, wherein the fourth layer is configured to serve as a        first tie layer; and    -   E) a fifth layer which is between the second layer and the third        layer, wherein the fifth layer is configured to serve as a        second tie layer.

In an exemplary embodiment the packaging article is selected from thegroup consisting of end-seal bag, side-seal bag, pouch, or backseamedpackaging article.

In an exemplary embodiment the preceding claims, wherein in an instancein which a defective seal is detected, the process further comprisinggenerating a signal comprising at least one member selected from thegroup consisting of an alarm, package flagging, displaying an image of adefective seal, generating a report, marking the packaging article, anddiverting the packaging article.

In an exemplary embodiment the at least a portion of the vision systemis contained within an enclosure configured to block at least 50% ofambient light.

In an exemplary embodiment the at least a portion of the vision systemis contained within an enclosure configured to block at least 85% ofambient light.

In an exemplary embodiment the film is a monolayer film.

In an exemplary embodiment the image data is selected from the groupconsisting of thermal image data, photoelasticity image data and ultraviolet fluorescence emitted image data.

In an exemplary embodiment in an instance in which a defective seal isdetected, the process further comprising generating a signal comprisingat least one member selected from the group consisting of an alarm,package flagging, displaying an image of a defective seal, generating areport, marking the packaging article, and diverting the packagingarticle.

In an exemplary embodiment the image data of the at least one seal areais captured by the image capture device at a speed of at least 5 imagesper second.

In an exemplary embodiment the image data of the at least one seal areais captured by the image capture device at a speed of at least 30 imagesper second.

In an exemplary embodiment the image data of the at least one seal areais captured by the image capture device at a speed of at least 100images per second.

In an exemplary embodiment the image data of the at least one seal areais captured by the image capture device at a speed of at least 250images per second.

In an exemplary embodiment the image data of the at least one seal areais captured by the vision system prior to pack off of a packagedproduct.

In an exemplary embodiment the vision inspection engine assigns a sealscore value to the image data of the seal area.

In an exemplary embodiment the vision inspection engine compares theseal score value of the image data of the seal area with a thresholdvalue.

In an exemplary embodiment the vision inspection engine sendsinstructions to a seal defect mechanism if the seal score value isbeyond a threshold value.

In an exemplary embodiment the vision inspection engine assigns aconfidence rating to the seal score value.

In one exemplary embodiment, a system for detecting a defective seal ofa packaging article is disclosed. The system for detecting a defectiveseal of a packaging article comprises

-   -   A) a sealing mechanism configured to seal a film to itself,        another film, or a packaging support to form a packaging article        by forming at least one seal area;    -   B) a vision system comprising an image capture device configured        to acquire image data of the at least one seal area with a        vision system;    -   C) assessing the image data of the seal area with a vision        inspection engine to verify the continuity of the seal, the        strength of the seal, or both the continuity and strength of the        seal area.

In an exemplary embodiment the sealing mechanism forms a heat seal.

In an exemplary embodiment the film comprises at least one layercontaining a fluorescence-based indicator.

In an exemplary embodiment the system further comprises a blue band passfilter.

In an exemplary embodiment the vision system is an ultraviolet visionsystem further comprising an ultraviolet light source.

In an exemplary embodiment the system further comprises:

-   -   A) exposing the packaging article to incident radiation to        excite the fluorescence-based indicator so that the        fluorescence-based indicator fluoresces;    -   B) acquiring image data of the fluorescence emitted from the        seal area by the packaging article, while the indicator is        fluorescing.

In an exemplary embodiment the fluorescence-based indicator comprises atleast one member selected from the group consisting ofultraviolet-indicator, infrared-indicator, dye, pigment, opticalbrightener, fluorescent whitening agent,2,2′-(2,5-thiophenylenediyl)bis(5-tert-butylbenzoxazole),hydroxyl-4-)p-tolylamino)anthracene-9,10-dione,2,5-thiophenediylbis(5-tert-butyl-1,3-benzoxazole), and anthraquinonedyestuff.

In an exemplary embodiment the vision inspection engine comprises acomputing apparatus comprising computer executable instructionsconfigured to determine whether fluorescent electromagnetic energyemitted by the excited fluorescence-based indicator is indicative of adefective seal.

In an exemplary embodiment the computer executable instructions comprisean artificial intelligent algorithm.

In an exemplary embodiment determining that the seal is defectivecomprises determining that the film exhibits a higher or lower intensityof fluorescence in a first region of the seal, relative to a secondregion of the seal.

In an exemplary embodiment determining that the seal is defectivecomprises determining that the film exhibits a higher or lower intensityof fluorescence in a first region of the seal, relative to an expectedlevel fluorescence.

In an exemplary embodiment determining that the seal is defectivecomprises determining at least one of (i) that the film exhibits ahigher or lower intensity of fluorescence in a first region of the seal,relative to a second region of the seal, (ii) that the film exhibits ahigher or lower intensity of fluorescence in a first region of the seal,relative to an expected level fluorescence, or (iii) both (i) and (ii).

In an exemplary embodiment the image data is time-delayed thermographydata captured at a time after the forming of at least one seal area.

In an exemplary embodiment the vision system is a thermography visionsystem comprising an infrared imaging device capable of capturing atemperature distribution based on the amount of infrared radiationemitted from the seal area.

In an exemplary embodiment the image data is taken between 2 and 30seconds after the forming at least one seal area.

In an exemplary embodiment the vision inspection engine comprises acomputing apparatus comprising computer executable instructionsconfigured to determine whether thermal image data captured of the sealis indicative of a defective seal.

In an exemplary embodiment determining that the seal is defectivecomprises determining that the film exhibits a higher or lower thermaltemperature in a first region of the seal, relative to a second regionof the seal.

In an exemplary embodiment the vision system is a photoelasticity visionsystem comprising:

-   -   (i) a first linear polarizer having a direction of polarization        oriented in a first direction;    -   (ii) a second linear polarizer have a direction of polarization        oriented orthogonal to the first direction;    -   (iii) a light source; and    -   (iv) an imaging device.

In an exemplary embodiment the image data is selected from the groupconsisting of thermal image data, photoelasticity image data and ultraviolet fluorescence emitted image data.

In an exemplary embodiment the image data of the at least one seal areais captured by the image capture device at a speed of at least 30 imagesper second.

In an exemplary embodiment the vision inspection engine assigns a sealscore value to the image data of the seal area.

In an exemplary embodiment the vision inspection engine compares theseal score value of the image data of the seal area with a thresholdvalue.

In an exemplary embodiment the vision inspection engine assigns aconfidence rating to the seal score value.

This brief description of the invention is intended only to provide abrief overview of subject matter disclosed herein according to one ormore illustrative embodiments, and does not serve as a guide tointerpreting the claims or to define or limit the scope of theinvention, which is defined only by the appended claims. This briefdescription is provided to introduce an illustrative selection ofconcepts in a simplified form that are further described below in thedetailed description. This brief description is not intended to identifykey features or essential features of the claimed subject matter, nor isit intended to be used as an aid in determining the scope of the claimedsubject matter. The claimed subject matter is not limited toimplementations that solve any or all disadvantages noted in thebackground.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the features of the invention can beunderstood, a detailed description of the invention may be had byreference to certain embodiments, some of which are illustrated in theaccompanying drawings. It is to be noted, however, that the drawingsillustrate only certain embodiments of this invention and are thereforenot to be considered limiting of its scope, for the scope of theinvention encompasses other equally effective embodiments. The drawingsare not necessarily to scale, emphasis generally being placed uponillustrating the features of certain embodiments of the invention. Inthe drawings, like numerals are used to indicate like parts throughoutthe various views. Thus, for further understanding of the invention,reference can be made to the following detailed description, read inconnection with the drawings in which:

FIG. 1 is a schematic view of a web production process for extruding anannular web to make an annular tape in accordance with some embodimentsherein;

FIG. 2 is a schematic view of a further web production process forconverting the annular tape produced in FIG. 1 into an annular filmtubing in accordance with some embodiments herein;

FIG. 3 is a block diagram of an in-line seal defect detection systemincluding a thermal vision system of an embodiment;

FIG. 4 is a block diagram of an in-line seal defect detection systemincluding a photoelasticity vision system of an embodiment;

FIG. 5 is a block diagram of an in-line seal defect detection systemincluding an ultraviolet vision system of an embodiment;

FIG. 6 is a block diagram of an in-line seal defect detection systemincluding a thermal vision system, a photoelasticity vision system andan ultraviolet vision system of an embodiment;

FIG. 7 is a block diagram of a seal defect detection system including athermal vision system, a photoelasticity vision system and anultraviolet vision system of an embodiment;

FIG. 8 is a schematic view of a thermography vision system of anembodiment;

FIG. 9 is a schematic view of a photoelasticity vision system of anembodiment;

FIG. 10 is a schematic view of the principle of operation of aphotoelasticity vision system according to an embodiment;

FIG. 11 is a schematic view of an ultraviolet vision system of anembodiment;

FIG. 12 is a block diagram of a vision system in accordance with anembodiment;

FIG. 13 is a block diagram of a system for assessing the integrity of aseal in accordance with an embodiment;

FIG. 14 is a block diagram of a system for assessing the integrity of aseal in accordance with an embodiment;

FIG. 15 is a flow sheet of an algorithm in accordance with anembodiment;

FIG. 16 is a flow sheet of a method of developing a trained imageclassification model in accordance with an embodiment;

FIG. 17 is a flow sheet of a method of developing a trained imageclassification model based on a number of parameters in accordance withan embodiment;

FIG. 18 is a flow sheet of a method of developing a trained imageclassification model based on a number of parameters in accordance withan embodiment;

FIG. 19 is a flow sheet of a method for an image classification systemto both train a model to classify seals and apply the artificialintelligent algorithm to classify states of a seal in accordance with anembodiment;

FIG. 20 is a schematic of a neural network that is a multilayer neuralnetwork in accordance with an embodiment;

FIG. 21 is a flow sheet of a method of classifying a state of a seal inaccordance with an embodiment;

FIG. 22 is a thermography images showing an acceptable seal;

FIG. 23 is an image views of thermography images showing a faulty seal;

FIG. 24 is an image view of a thermography image showing a faulty seal;

FIG. 25 is an image view of a thermography image showing acceptableseals;

FIG. 26 is an image view of a thermography image showing acceptableseals;

FIG. 27 is an image view of a thermography image showing faulty sealareas;

FIG. 28 is an image view of a thermography image showing faulty sealareas;

FIG. 29 is an image view of a photoelasticity image showing anacceptable seal;

FIG. 30 is an image view of a photoelasticity image showing a faultyseal;

FIG. 31 is an image view of a photoelasticity image showing a faultyseal; and

FIG. 32 is an image view of an ultraviolet image showing a faulty seal.

DETAILED DESCRIPTION

Example embodiments will now be described more fully hereinafter withreference to the accompanying drawings, in which some, but not allembodiments are shown. Indeed, the embodiments may take many differentforms and should not be construed as limited to the embodiments setforth herein; rather, these embodiments are provided so that thisdisclosure will satisfy applicable legal requirements. Like referencenumerals refer to like elements throughout. The terms “data,” “content,”“information,” and similar terms may be used interchangeably, accordingto some example embodiments, to refer to data capable of beingtransmitted, received, operated on, and/or stored. Moreover, the term“exemplary,” as may be used herein, is not provided to convey anyqualitative assessment, but instead merely to convey an illustration ofan example. Thus, use of any such terms should not be taken to limit thespirit and scope of embodiments of the present invention.

Referring to FIGS. 1-2, an exemplary embodiment of forming a packagingarticle is shown. In some embodiments, the packaging article may bemanufactured using any known method provided an indicator as describedherein is incorporated into the packaging article. In some embodiments,solid polymer beads (not illustrated) are fed to a plurality ofextruders 28 (for simplicity, only one extruder is illustrated). Insideextruders 28, the polymer beads are forwarded, melted, and degassed,following which the resulting bubble-free melt is forwarded into diehead 30, and extruded through an annular die, resulting in annular tape32.

After cooling and quenching by water spray from cooling ring 34, annulartape 32 is collapsed into lay-flat configuration by nip rollers 36.Annular tape 32 in lay-flat configuration is then passed throughirradiation vault 38 surrounded by shielding 40, where annular tape 32is irradiated with high energy electrons (i.e., ionizing radiation) fromiron core transformer accelerator 42. Annular tape 32 is guided throughirradiation vault 38 on rolls 44.

After irradiation, irradiated annular tape 46 is directed throughpre-coating nip rollers 48, following which irradiated annular tape 46is slightly inflated, resulting in trapped bubble 50. At trapped bubble50, irradiated annular tape 46 is not significantly drawnlongitudinally, as the surface speed of post-coating nip rollers 52 isabout the same as the surface speed of pre-coating nip rollers 48.Furthermore, irradiated tape 46 is inflated only enough to place theannular tape into a substantially circular configuration withoutsignificant transverse orientation, i.e., without transverse stretching.

Irradiated annular tape 46, slightly inflated by bubble 50, is passedthrough vacuum chamber 54, and thereafter forwarded through coating die56. Annular coating stream 58 is melt extruded from coating die 56 andcoated onto inflated, irradiated annular tape 46, to form coated annulartape 60. Coating stream 58 comprised an O₂-barrier layer made from PVDC,together with additional layers, all of which do not pass through theionizing radiation. Further details of the above-described coating stepare generally as set forth in U.S. Pat. No. 4,278,738, to BRAX et. al.,which is hereby incorporated by reference thereto, in its entirety.After irradiation and coating, coated annular tape 60, is wound up ontowindup roll 62.

Thereafter, turning to FIG. 2, windup roll 62 is installed as unwindroll 64, on a second stage in the process of making the desiredheat-shrinkable film tubing. Coated annular tape 60 is unwound fromunwind roll 64, and passed over guide roll 66, after which coatedannular tape 60 is passed into hot water bath tank 68 containing hotwater 70. Coated tubular film 60, still in lay-flat configuration, isimmersed in hot water 70, long enough to bring annular tape 60 up to itssoftening point, i.e., the desired temperature for biaxial orientationwhile the coated annular tape is in the solid state.

Thereafter, coated annular tape 60 is directed through nip rolls 72, andbubble 74 is blown, thereby transversely solid state stretching coatedannular tape 60. Furthermore, while being blown, i.e., transverselystretched, nip rolls 76 draws annular tape 60 in the longitudinaldirection, as nip rollers 76 having a surface speed higher than thesurface speed of nip rollers 72. As a result of the transversestretching and longitudinal drawing, annular tape 60 is biaxiallyoriented in the solid state to form biaxially-oriented, heat-shrinkablefilm tubing 78. Heat-shrinkable film tubing 78 is stretchedtransversely, and drawn longitudinally. While bubble 74 is maintainedbetween pairs of nip rollers 72 and 76, the resulting blown film tubing78 is collapsed into lay-flat configuration by rollers 80. Film tubing78 in lay-flat configuration is thereafter conveyed through nip rollers76 and across guide roll 82, and then rolled onto wind-up roll 84. Idlerroll 86 assures a good wind-up.

As used herein, the term “film” is inclusive of plastic web, regardlessof whether it is a film, bag or sheet. Multi-layer films as disclosedherein may have a total thickness of up to 0.15 mm. In an embodiment,the multilayer film comprises: A) a first layer which is a first outerfilm layer and which serves as a heat seal layer; B) a second layerwhich is a second outer layer and which serves as an abuse layer; C) athird film layer which is between the first layer and the second layer,the third layer serving as a functional layer; D) a fourth film layerwhich is between the first layer and the third layer, the fourth layerserving as a first tie layer; and E) a fifth film layer which is betweenthe second layer and the third layer, the fifth layer serving as asecond tie layer. In some embodiments the film may comprise a firstlayer, second layer, and third layer as described above, without thefirst and second tie layers. Function layers include, but are notlimited to an oxygen barrier layer, organoleptic barrier layer, moisturebarrier layer, hazardous chemical barrier layer, microbial barrierlayer, acid layer, acid salt layer, bacteriocin layer, bacteriophagelayer, metal layer, metal salt layer, natural oil layer, natural extractlayer, layer containing polyhexamethylene biguanide hydrochloride, layercontaining paraben, layer containing grafted silane-quaternary amine,layer containing triclosan, layer containing zeolite of silver, copper,and/or zinc.

In an embodiment, the film is adhered to a packaging support such as atray or another film to form a packaging article. In an embodiment, thepackaging article is a member selected from the group consisting ofend-seal bag, side-seal bag, pouch, or backseamed packaging article. Insome embodiments, the packaging article may be a film container like abag (e.g., end-seal bag, side-seal bag, etc.) having film surroundingthe product. The film may include substantially uniformly distributedfluorescent indicator therein such that the packaged product mayindicate gas within the packaging article on all sides of the product.In some embodiments, the packaging article may include a rigid orsemi-rigid shell with a film covering or seal enclosing the shell. In anembodiment, the product placed in the packaging article is a foodproduct. In an embodiment, the food product comprises at least onemember selected from the group consisting of meat and cheese.

Referring now to FIG. 3 is a block diagram of an exemplary process flowfor detecting defects in the seal area of a packaging article. A film orpackaging article is sealed by a sealing mechanism 31 to create a sealarea by sealing the film to itself or another packaging support. In someembodiments, the packaging article may include a film, a pouch, a bag,tubing, rigid or semi-rigid shell or tray with a film covering or sealenclosing the shell or tray. In some embodiments, the packaging articlemay include a skin pack. The sealing mechanism 31 creates a seal area inthe film. The sealing mechanism 31 is a heat generating seal mechanismsuch as a heating from a seal bar, impulse seal, ultrasonic seal, highspeed heat sealing (sealing at least 100 seals per minute) or laserseal. The sealing mechanism seals the film to itself. In the instance ofa film tubing as described above, the sealing mechanism seals one end ofthe tubing to create a packaging article with one opening. It isunderstood that the sealing mechanism could also seal the sides of thepackaging article and may also seal the final opening after a producthas been placed in the packaging article.

After sealing of the packaging article, a thermography vision system 33captures thermographic images of the seal area. Residual heat from thesealing process performed by the sealing mechanism 31 is retained in theseal area. This allows for the capture of thermographic images of theseal area. The thermal image is captured between 2 and 40 seconds aftersealing. In an embodiment the thermal image is captured between 5 and 30seconds after sealing. In an embodiment the thermal image is capturedbetween 10 and 20 seconds after sealing. In an embodiment the thermalimage is captured about 15 seconds after sealing. In an embodiment thethermal image is captured about 20 seconds after sealing. It isunderstood that active cooling systems such as blown air, chilled air,chill chambers and the like can be used to reduce image capture timeframes.

Thermography is a contactless, non-destructive testing method in whichsurface temperature distribution (thermograms) are recorded based on theamount of infrared radiation emitted by the inspected scene. Theinspected objects are thermally excited, and the response to thisexcitation is recorded in time. Active thermography data arethree-dimensional since the temperature of each spatial pixel isevaluated in time. Active thermography is used to detect (sub)surfacedefects based on differences in thermo-physical properties.

The thermography vision system 33 includes at least one thermographyimage capture device and a vision inspection engine. In some embodimentsthe thermography vision system 33 includes a light source, a number oflight sources (of the same or different types), filters, multiple imagecapture devices or reflectors. In an embodiment the thermography imagecapture device is a thermal camera. The light source may be a continuouslight source or a flashed light source. The light source may be whitelight, filtered light, diffused light, infrared light, ultravioletlight, in the visible spectrum or outside of the visible spectrum. Insome embodiments the thermography vision system 33 is housed in anenclosure to provide an isolated environment for the image capturedevice to capture images. The enclosure may isolate or reduce the effecton the imaging area from external sources of light and containments suchas dust.

In one embodiment the thermography image capture device captures animage of at least one seal of each packaging article. In anotherembodiment the thermography image capture device captures at least twoimages of at least one seal of each packaging article. In anotherembodiment the thermography image capture device captures at least fourimages of at least one seal of each packaging article. In one embodimentthe thermography image capture device captures image data at a framerate of at least 20 frames per second. In one embodiment thethermography image capture device captures image data at a frame rate ofat least 40 frames per second. In one embodiment the thermography imagecapture device captures image data at a frame rate of at least 60 framesper second. Having additional image data of the seal provides additionaldata and may result in more reliable determinations of seal quality. Inan embodiment, the thermography image capture device transmits imagedata in a video format, including but not limited to motion JPEG, MPEG,SEQ and H.264 format. In an embodiment, the thermography image capturedevice captures thermal image data. In another embodiment, thethermography image capture device captures multi-spectral dynamic imagedata.

The image data acquired by the thermal imaging device is sent to avision inspection engine. The vision inspection engine, as described inmore detail herein, is capable of analyzing images and determining if aseal is acceptable of if a seal has a defect. The vision inspectionengine creates a seal score relating to the quality of the seal area.The vision inspection engine analyzes at least one of color gradientchange, light intensity, color intensity, change in data from adjacentpixels, change in data from adjacent sections of pixels and combinationsthereof to create a seal score. In an embodiment, the vision inspectionengine analyzes at least one of color gradient change, light intensity,color intensity, change in data from adjacent pixels, change in datafrom adjacent sections of pixels and combinations thereof to furthercreate a confidence score. The confidence score provides an assessmentof the reliability of the seal score assigned. The vision inspectionengine may be local to the thermography visions system 33, integratedinto another part of a system or be remote.

Still referring to FIG. 3, defect mechanism 35 receives instructions toidentify a packaging article having a defective seal, or having aconfidence rating below a certain threshold. In one embodiment thedefect mechanism pulls the defective packaging article from the line foroff-line testing, verification, recycling, repacking or waste therebyonly allowing packaging articles with adequate seals to continue beyondthe defect mechanism 35. In an embodiment the defect mechanism tags,flags or prints at least one indicium on the packaging article toidentify the packaging article as potentially having a defective seal.In an embodiment, the detection of a defective seal may result in thedefect mechanism activating an alarm, flagging the defective packagingarticle, displaying an image of the defective seal or packaging article,displaying data pertaining to the defective seal, and generating areport of the defective seal or packaging article. In an embodiment, thedefect mechanism removes the packaging article having a defective seal.The defect mechanism may pause the production line while the packagingarticle is off loaded, or it may remove the packaging article withoutpausing the production line. In an embodiment, the defect mechanismincludes vacuum suction cups utilized in combination with robotic armsto automatically remove package articles. In another embodiment, thedefect mechanism moves the packaging to a defect line away from the linepackaging articles not having a defective seal.

In an embodiment, a signal in response to the defective seal isgenerated and activates the alarm, flagging, discontinuity imagedisplay, discontinuity data, report of discontinuity data, etc. whilethe web remains in motion, i.e., instantaneously and online.Alternatively, the signal in response to the defective seal is generatedafter production is complete, i.e., offline. The signal in response tothe discontinuity can include electronic messaging, email, data log, andreport.

It is understood that the vision system and defect mechanism may beimplemented after the formation of the packaging article and beforeproduct packaging. In another embodiment, the vision system and defectmechanism may be implemented after the packaging and sealing of aproduct in the packaging article.

Referring now to FIG. 4 is a block diagram of an exemplary process flowfor detecting defects in the seal area of a packaging article. A film orpackaging article is sealed by a sealing mechanism 31 as describedabove. The sealing mechanism 31 creates a seal area in the film. Thesealing mechanism 31 may be a seal bar heat seal, impulse seal,ultrasonic seal, high speed heat sealing or laser seal. In oneembodiment, the sealing mechanism creates at least 100 seals per minute.In another embodiment, the sealing mechanism creates at least 300 sealsper minute. The sealing mechanism seals the film to itself. In theinstance of a film tubing as described above, the sealing mechanismseals one end of the tubing to create a packaging article with oneopening. It is understood that the sealing mechanism could also seal thesides of the packaging article and may also seal the final opening aftera product has been placed in the packaging article.

The film is positioned between two cross-polarized filters setorthogonal to each other. The first filter passes light oriented in afirst direction. The second filter is oriented orthogonal to the firstfilter to block all light coming through the first filter. When a pieceof transparent material is positioned between the two filters, the lightis rotated. The amount of rotation varies with the type of material andthe amount of internal strain within the material. With the lightrotated out of the polarization plane of the second filter, some lightpasses through the second filter.

After sealing of the packaging article, a photoelasticity vision system43 captures images of the seal area. The photoelasticity vision system43 includes at least one photoelasticity image capture device, a lightsource, a first polarizing light filter, a second polarizing lightfilter and a vision inspection engine. In some embodiments thephotoelasticity vision system 43 includes a number of light sources (ofthe same or different types), filters, multiple image capture devices orreflectors. In an embodiment the photoelasticity image capture device isa high speed camera. The light source may be a continuous light sourceor a flashed light source. The light source may be white light, filteredlight, diffused light, infrared light, ultraviolet light, in the visiblespectrum or outside of the visible spectrum. In some embodiments thephotoelasticity vision system 43 is housed in an enclosure to provide anisolated environment for the image capture device to capture images. Theenclosure may isolate or reduce the effect on the imaging area fromexternal sources of light and containments such as dust.

In one embodiment the photoelasticity image capture device captures animage of at least one seal of each packaging article. In anotherembodiment the photoelasticity image capture device captures at leasttwo images of at least one seal of each packaging article. In anotherembodiment the photoelasticity image capture device captures at leastfour images of at least one seal of each packaging article. In oneembodiment the photoelasticity image capture device captures image dataat a frame rate of at least 20 frames per second. In one embodiment thephotoelasticity image capture device captures image data at a frame rateof at least 40 frames per second. In one embodiment the photoelasticityimage capture device captures image data at a frame rate of at least 60frames per second. Having additional image data of the seal providesadditional data and may result in more reliable determinations of sealquality. In an embodiment, the ultraviolet image capture device is anRGB camera capable of capturing images in the visible spectrum.

The image data acquired by the photoelasticity imaging device is sent toa vision inspection engine. The vision inspection engine, as describedin more detail herein, is capable of analyzing images and determining ifa seal is acceptable of if a seal has a defect. In an embodiment, thevision inspection engine determines a confidence rating along with theseal classification. The vision inspection engine may be local to thephotoelasticity vision system 43, integrated into another part of asystem or be remote. The vision inspection engine analyzes the image forvariations in adjacent pixels color gradients, section comparison andthe like. The vision inspection engine may compare a number of pixels,or sections of images one or more of the following: mean, variable,skew, minimum, maximum, range or variations in the seal area. Variationsbetween pixels or section of the image can be indicative of a sealdefect.

Still referring to FIG. 4, defect mechanism 35 receives instructions toidentify a packaging article having a defective seal or having aconfidence rating below a certain threshold. In one embodiment thedefect mechanism pulls the defective packaging article from the line foroff-line testing, verification, recycling, repacking or waste therebyonly allowing packaging articles with adequate seals to continue beyondthe defect mechanism 35. In an embodiment the defect mechanism tags,flags or prints at least one indicium on the packaging article toidentify the packaging article as potentially having a defective seal.In an embodiment, the detection of a defective seal may result in thedefect mechanism activating an alarm, flagging the defective packagingarticle, displaying an image of the defective seal or packaging article,displaying data pertaining to the defective seal, and generating areport of the defective seal or packaging article.

In an embodiment, a signal in response to the defective seal isgenerated and activates the alarm, flagging, discontinuity imagedisplay, discontinuity data, report of discontinuity data, etc. whilethe web remains in motion, i.e., instantaneously and online.Alternatively, the signal in response to the defective seal is generatedafter production is complete, i.e., offline. The signal in response tothe discontinuity can include electronic messaging, email, data log, andreport.

It is understood that the vision system and defect mechanism may beimplemented after the formation of the packaging article and beforeproduct packaging. In another embodiment, the vision system and defectmechanism may be implemented after the packaging and sealing of aproduct in the packaging article.

Referring now to FIG. 5 is a block diagram of an exemplary process flowfor detecting defects in the seal area of a packaging article. A film orpackaging article is sealed by a sealing mechanism 31. In someembodiments, the packaging article may include a film, a pouch, a bag,tubing, rigid or semi-rigid shell with a film covering or seal enclosingthe shell. In some embodiments, the film may include a detectablecomponent allowing detection seal defects. In some embodiments, thepackaging article may include a skin pack. In some embodiments, the filmhaving an ultraviolet (“UV”) fluorescence-based indicator may onlypartially surround the product.

The films include a detectable component such as a fluorescence-basedindicator. In an embodiment, the film contains a layer comprising ablend of a polymer and the fluorescence-based indicator in anyfunctional layer in which the concentration of the fluorescent-basedindicator is present in the blend in an amount of from about 5 to 50ppm, based on total layer weight, with the fluorescence-based indicatorbeing uniformly blended with the polymer. The uniformity of theindicator in the indicator/polymer blend refers to the concentration ofthe indicator in the blend being subject to a variation of not more than20%, and not less than 20%, based on a target concentration of theindicator in the blend, upon taking 10 random samples, each samplehaving a size of 10 grams. In an embodiment, the polymer comprises PVDC,i.e., polyvinylidene chloride, ethylene vinyl alcohol or nylon.

The UV fluorescent indicator can be present in the functional layer atany level that is detectable by the detector while allowing thefunctional layer to maintain its intended function. Too much fluorescentindicator may interfere with layer function. Too little fluorescentindicator may become undetectable to the detector. In an embodiment, thefluorescent indicator may be present at a level of at least 0.5 partsper million (ppm). As used herein, the phrase “part per million” and theequivalent expression “ppm” refer to the weight of the fluorescentindicator versus the total weight of the layer (weight fluorescentindicator+weight of remainder of components in the layer). Of course,the majority component of the layer is one or more thermoplasticpolymers which are a solid at room temperature. Both the fluorescentindicator and the thermoplastic polymer of the layer can be solids atroom temperature. In an embodiment, the fluorescent indicator can bepresent at a level of at least 1 ppm, or at least 1.5 ppm, or at least 2ppm, or at least 3 ppm, or at least 5 ppm, or at least 10 ppm, or atleast 20 ppm, or at least 40 ppm, or at least 80 ppm, or at least 120ppm, or at least 160 ppm, or at least 200 ppm, or at least 300 ppm, orat least 500 ppm. In an embodiment, the fluorescent indicator can bepresent in the layer at a level of from 0.5 to 40 ppm, or from 1 to 20ppm, or from 1.5 to 10 ppm, or from 2 to 5 ppm. In order for a film tobe suitable for food contact end use, the fluorescent indicator ispresent in the layer in an amount of not more than 150 ppm.

A UV-based fluorescent indicator is a UV-absorbing compound withdistinctive absorption and/or fluorescence properties. PreferredUV-absorbing fluorescent indicator component has a unique opticalsignature that is not present in nature and not easily confused withsignals from natural sources. A preferred UV-fluorescent indicator hasmultiple unique absorption or fluorescent features in its UV spectra.For example, as used herein, electromagnetic radiation at 375 nanometerswas used as incident radiation to excite a fluorescent indicator knownas 2,5-thiophenediylbis(5-tert-butyl-1,3-benzoxazole), which is assignedCAS registry number 7128-64-5, and which is also known as:2,2′-(2,5-thiophenediyl)bis[5-tert-butylbenzoxazole];2,5-bis-2(5-tert-butyl-benzoxalyl)thiophene;2,5-bis(5-t-butyl-2-benzoxazolyl)thiophene;2,5-bis-(5-t-butylbenzoxazolyl-[2-yl])-thiophene;2,5-bis-(5-tert-butyl-2-benzoxazol-2-yl)thiophene;2,5-bis(5′-tert-butyl-2-benzoxazol-2-yl)thiophene;2,5-bis(5′-tert-butyl-2′-benzoxazolyl)thiophene;2,5-bis(5-tert-butyl-2-benzoxazolyl)thiophene;2,5-bis(5-tert-butyl-benzoxazol-2-yl)thiophene;2,5-bis(5-tert-butylbenzoxazoyl)-2-thiophene;2,5-di(5-tertbutylbenzoxazol-2-yl)thiophene;2,2′-(2,5-thiophenediyl)bis[5-(1,1-dimethylethyl)-benzoxazole;2,5-bis(5′-tert-butyl-2-benzoxazolyl)thiophene; and2,5-thiophenediylbis(5-tert-butyl-1,3-benzoxazole). The absorption ofthe incident radiation at 375 nanoimeters caused the excited2,5-thiopbenediylbis(5-tert-butyl-1,3-benzoxazole) optical brightenerdetectable component to emit radiation at 435 nanometers. Thefluorescent indicator was uniformly blended into a PVDC resin which wasused to produce an oxygen barrier layer of a multilayer film. Exposingthe resulting annular tape and/or heat-shrinkable film tubing toincident radiation at 375 nm excited the2,5-thiophenediylbis(5-tert-butyl-1,3-benzoxazole) optical brightener toemit radiation at 435 nanometers. The emitted 435 nm radiation wasdetected by a machine vision system, which revealed the presence,continuity, and thickness of the PVDC barrier layer of the tape and amultilayer film tubing. In an embodiment, the UV-based fluorescentindicator comprises at least one member selected from the groupconsisting of ultraviolet-indicator, infrared-indicator, dye, pigment,optical brightener, fluorescent whitening agent,2,2′-(2,5-thiophenylenediyl)bis(5-tert-butylbenzoxazole),hydroxyl-4-)p-tolylamino)anthracene-9,10-dione,2,5-thiophenediylbis(5-tert-butyl-1,3-benzoxazole), and anthraquinonedyestuff. The indicator is of a type which, if exposed to radiation at afirst peak wavelength, emits radiation at a second peak wavelength.

In an embodiment, the detectable component comprises at least one memberselected from the group consisting of ultraviolet-indicator,infrared-indicator, dye, pigment, optical brightener, fluorescentwhitening agent, and 2,5-thiophenediylbis(5-tert-butyl-1,3-benzoxazole),2,2′-(2,5-thiophenylenediyl)bis(5-tert-butylbenzoxazole),hydroxyl-4-)p-tolylamino)anthracene-9,10-dione,2,5-thiophenediylbis(5-tert-butyl-1,3-benzoxazole), and anthraquinonedyestuff. 2,5-Thiophenediylbis(5-tert-butyl-1,3-benzoxazole) is marketedas an optical brightener by a plurality of suppliers, including BASFCorporation (TINOPAL OP®2,5-thiophenediylbis(5-tert-butyl-1,3-benzoxazole) fluorescentbrightening agent) and Mayzo, Inc (BENETEX OB PLUS®2,5-thiophenediylbis(5-tert-butyl-1,3-benzoxazole) fluorescentbrightening agent). The indicator is of a type which, if exposed toradiation at a first peak wavelength, emits radiation at a second peakwavelength.

In an embodiment, the UV-based fluorescent indicator is present in atleast one layer of the film, with the indicator being present at a levelof from 0.5 to 150 ppm, based on layer weight. In another embodiment,the indicator is present in at least one layer of the film, with theindicator being present at a level of from 1 to 20 ppm, or from 2 to 10ppm, based on layer weight. In an embodiment, the detectable componentis present in a film layer at a level of at least 1 part per million.

In an embodiment, the UV-based fluorescent indicator is of a type which,if exposed to radiation at a first peak wavelength, emits radiation at asecond peak wavelength.

Still referencing to FIG. 5, the sealing mechanism 31 creates a sealarea in the film. The sealing mechanism 31 seals the film to itself oranother packaging article. In an embodiment, the sealing mechanism is aheat generating seal mechanism such as a heating from a seal bar,impulse seal, ultrasonic seal or laser seal. The sealing mechanism sealsthe film to itself. In the instance of a film tubing as described above,the sealing mechanism seals one end of the tubing to create a packagingarticle with one opening. It is understood that the sealing mechanismcould also seal the sides of the packaging article and may also seal thefinal opening after a product has been placed in the packaging article.

After sealing of the packaging article, an ultraviolet system 53captures ultraviolet images of the seal area. The ultraviolet visionsystem 53 includes at least one ultraviolet image capture device, anultraviolet light source and a vision inspection engine. In anembodiment, the ultraviolet image capture device is an RGB cameracapable of capturing images in the visible spectrum. In some embodimentsthe ultraviolet vision system 53 includes a number of light sources (ofthe same or different types), filters, multiple image capture devices orreflectors. The light sources may be a continuous light source or aflashed light source. The additional light source may be white light,filtered light, diffused light, infrared light, ultraviolet light, inthe visible spectrum or outside of the visible spectrum. In someembodiments the ultraviolet vision system 53 is housed in an enclosureto provide an isolated environment for the image capture device tocapture images. The enclosure may isolate or reduce the effect on theimaging area from external sources of light and containments such asdust. The light source of the ultraviolet vision system 53 excites adetectable component in the seal area of the packaging article. Thedetectable component allows the image capture device to identify thedetectable component.

In an embodiment, the detectable component is a UV fluorescent indicatorand at least one of (i) a fluorescent electromagnetic energy intensityis higher in a region where a seal defect is located as compared to anadequate seal, or (ii) a fluorescent electromagnetic energy color shift(downward in energy, i.e., longer wavelength, lower frequency) occurs ina region where a seal defect is located as compared to an adequate seal.

In an embodiment, the fluorescent electromagnetic energy intensity is atleast 10% higher in the region where a seal defect is located asrelative to an adequate seal. In another embodiment, the fluorescentelectromagnetic energy intensity is >15% higher, or >20% higher, or >30%higher, or >40% higher, or >50% higher, or >60% higher in the regionwherein a seal defect is located, relative to the intensity of thefluorescent electromagnetic energy in a region in which seal isadequate. Factors that can affect the fluorescence image emitted by thepackage include, among others: (i) angle of view, (ii) focal distancebetween imaging device and area of package under consideration, (iii)exposure time, (iv) amount of excitation, and (v) thickness variationsin the film, including the increase in thickness at a heat seal of thefilm to itself or to another film.

In one embodiment the ultraviolet image capture device captures an imageof at least one seal of each packaging article. In another embodimentthe ultraviolet image capture device captures at least two images of atleast one seal of each packaging article. In another embodiment theultraviolet image capture device captures at least four images of atleast one seal of each packaging article. In one embodiment theultraviolet image capture device captures image data at a frame rate ofat least 20 frames per second. In one embodiment the ultraviolet imagecapture device captures image data at a frame rate of at least 40 framesper second. In one embodiment the ultraviolet image capture devicecaptures image data at a frame rate of at least 60 frames per second.Having additional image data of the seal provides additional data andmay result in more reliable determinations of seal quality.

The image data acquired by the ultraviolet imaging device is sent to avision inspection engine. The vision inspection engine, as described inmore detail herein, is capable of analyzing images and determining if aseal is acceptable of if a seal has a defect. In an embodiment, theacquiring of the image data is carried out using an ultraviolet imagingdevice that generates image data of the fluorescent electromagneticenergy emitted by the excited indicator, with the assessing of the imagedata being carried out using a vision inspection engine programmed withan algorithm capable of assessing intensity of the fluorescentelectromagnetic energy emitted by the excited indicator. In anembodiment, the vision inspection engine determines a confidence ratingalong with the seal classification. The vision inspection engine may belocal to the ultraviolet visions system 33, integrated into another partof a system or be remote.

Still referring to FIG. 5, defect mechanism 35 receives instructions toidentify a packaging article having a defective seal, or having aconfidence rating below a certain threshold. In one embodiment thedefect mechanism pulls the defective packaging article from the line foroff-line testing, verification, recycling, repacking or waste therebyonly allowing packaging articles with adequate seals to continue beyondthe defect mechanism 35. In an embodiment the defect mechanism tags,flags or prints at least one indicium on the packaging article toidentify the packaging article as potentially having a defective seal.In an embodiment, the detection of a defective seal may result in thedefect mechanism activating an alarm, flagging the defective packagingarticle, displaying an image of the defective seal or packaging article,displaying data pertaining to the defective seal, and generating areport of the defective seal or packaging article.

In an embodiment, a signal in response to the defective seal isgenerated and activates the alarm, flagging, discontinuity imagedisplay, discontinuity data, report of discontinuity data, etc. whilethe web remains in motion, i.e., instantaneously and online.Alternatively, the signal in response to the defective seal is generatedafter production is complete, i.e., offline. The signal in response tothe discontinuity can include electronic messaging, email, data log, andreport.

It is understood that the vision system and defect mechanism may beimplemented after the formation of the packaging article and beforeproduct packaging. In another embodiment, the vision system and defectmechanism may be implemented after the packaging and sealing of aproduct in the packaging article.

In some embodiments, multiple vision systems are envisioned. The visionsystems may be redundant as a similar type of vision system or distinctvision systems utilizing distinct light sources, image capture devices,lighting angles, lighting properties and the like. Referring now to FIG.6 is a block diagram of an exemplary process flow for detecting defectsin the seal area of a packaging article. The process is similar to theembodiments described above in reference to FIGS. 3-5 with the additionof additional vision systems included. The thermography vision system33, photoelasticity vision system 43 and ultraviolet vision system 53each send image data to a vision inspection engine. It is understoodthat a common shared vision inspection engine or multiple visioninspections engines are contemplated. The vision inspection engine isprogrammed with an algorithm or algorithms capable of assessingthermography images, photoelasticity images and intensity of thefluorescent electromagnetic energy emitted by the excited indicator. Thecombined image data from the three vision systems result in a higherconfidence rating which is sent to the defect mechanism 35. While athermography vision system 33, photoelasticity vision system 43 andultraviolet vision system 53 are shown, a system is contemplated thatutilizes only two vision systems.

Referring now to FIG. 7 is a block diagram of an exemplary process flowfor detecting defects in the seal area of a packaging article after aproduct is sealed within the packing article. The product loadingmechanism loads a product into a packaging article. In one embodiment,the product being a food product. In another embodiment, the productbeing meet or cheese. After the product is loaded, the sealing mechanism31 seals the product in the packaging article. In some embodiments, thesealing mechanism includes a vacuum chamber to vacuum seal the productin the packaging article. In an embodiment, the packaging article isshrunk around the product. In an embodiment, the packaging article is anon-shrink packaging article. The seal or seals of the packaging articleare then analyzed by the thermography vision system 33, photoelasticityvision system 43 and ultraviolet vision system 53 and vision inspectionengine(s) as described more fully herein. The combined image data fromthe three vision systems result in a higher confidence rating which issent to the defect mechanism 35. While a thermography vision system 33,photoelasticity vision system 43 and ultraviolet vision system 53 areshown, a system is contemplated that utilizes only two vision systems.

Turning now to FIG. 8, there is shown a thermography vision system 81according to one embodiment. Film 100 is transported to sealingmechanism 85 to seal the film 100 to itself or to another article. Inone embodiment the sealing mechanism creates at least 10 seals perminute. In an embodiment, the sealing mechanism creates at least 50seals per minute. In an embodiment, the sealing mechanism creates atleast 100 seals per minute. In an embodiment, the sealing mechanismcreates at least 250 seals per minute. In one embodiment, the sealingmechanism 85 is a heat generating seal mechanism such as a seal barsealer, impulse seal, ultrasonic seal, or laser seal.

After sealing, the film 100 continues beyond the sealing mechanism 85where a thermography image capture device 87 captures at least one imageof the seal created by the sealing mechanism 85. The thermography imagecapture device is an infrared scanner or camera. In an embodiment, theinfrared camera is capable of capturing at least two images of each sealas the film is in motion. In an embodiment, the infrared camera iscapable of capturing at least four images of each seal as the film is inmotion. In an embodiment, the infrared camera is capable of capturingimages of each seal at a speed of at least 30 frames per second. In anembodiment, the infrared camera is capable of capturing images of eachseal at a speed of at least 60 frames per second. In an embodiment thethermography image capture device 87 is positioned to capture an imageof a seal between 2-40 seconds after the seal is created by the sealingmechanism 85 in the film 100. In an embodiment the thermography imagecapture device 87 is positioned to capture an image of a seal 5-30seconds after the seal is created by the sealing mechanism 85 in thefilm 100. In an embodiment the thermography image capture device 87 ispositioned to capture an image of a seal 10-20 seconds after the seal iscreated by the sealing mechanism 85 in the film 100. In an embodimentthe thermography image capture device 87 is positioned to capture animage of a seal about two seconds after the seal is created by thesealing mechanism 85 in the film 100. In an embodiment the thermographyimage capture device 87 is positioned to capture an image of a sealabout five seconds after the seal is created by the sealing mechanism 85in the film 100. In an embodiment the thermography image capture device87 is positioned to capture an image of a seal about 10 seconds afterthe seal is created by the sealing mechanism 85 in the film 100. In anembodiment the thermography image capture device 87 is positioned tocapture an image of a seal about 20 seconds after the seal is created bythe sealing mechanism 85 in the film 100. The residual heat from thesealing mechanism 85 allows for the thermography image capture device 87to capture a thermal image of the seal along with variations intemperature along the seal area. Such variations may be indicative of adefective seal. The image data captured by the thermal image capturedevice 87 is sent to a vision inspection engine 101 as described in moredetail herein.

Referring now to FIG. 9 there is shown a photoelasticity vision system91 according to one embodiment. Film 100 is transported to a positionwhere the photoelasticity image capture device 96 and capture an imageof light that passes through the seal area of film 100, linearpolarizers 94 and 95. In an embodiment, the photoelasticity imagecapture device is an RGB camera capable of capturing images in thevisible spectrum. The light from light source 91 passes through diffuseplate 93 to diffuse the light a first linear polarizer 94 having adirection of polarization oriented in a first direction to polarize thelight from light source 91. It is understood that a diffused lightssource can be utilized in lieu of, or in addition to use of a diffuseplate. The polarized light then passes through film 100 causing thelight to rotate. The amount of rotation varies based on variables suchas type of film, seal strength and seal integrity. With some of thelight rotated, that light then passes through a second linear polarizer95 having a direction of polarization oriented orthogonal to the firstdirection of the first linear polarizer 94. In the absence of film 100,no light would be transmitted from the light source 91 to thephotoelasticity image capture device 96. However, since the film 100rotates the light, photoelasticity image capture device 96 captures animage of the film 100.

In an embodiment, the light source 91 is a white light source having awavelength spectrum from 400 nm-700 nm. In an embodiment the lightsource is a background light below the film. In another embodiment thelight source is a polarized white light source, eliminating the need forthe first linear polarizer.

While two linear polarizers are described above, it is contemplated thatat least one of the linear polarizers is replaced by utilizing apolarizing filter film on the light source or the lens of thephotoelasticity image capture device. In another embodiment, thephotoelasticity image capture device includes a polarized lens toclimate the need for the second linear polarizer. In another embodiment,the light source is a polarized light source. In an embodiment, thephotoelasticity camera is capable of capturing at least two images ofeach seal as the film is in motion. In an embodiment, thephotoelasticity camera is capable of capturing at least four images ofeach seal as the film is in motion. In an embodiment, thephotoelasticity camera is capable of capturing images of each seal at aspeed of at least 30 frames per second. In an embodiment, thephotoelasticity camera is capable of capturing images of each seal at aspeed of at least 60 frames per second.

The image data captured by the photoelasticity image capture device 97is sent to a vision inspection engine 101 as described in more detailherein.

Referring now to FIG. 10 a schematic view the photoelasticity operation.The first polarizing filter 1002 has a direction of polarizationorthogonal to that of the second polarizing filter 1004. Nonpolarizedlight 1001 passes through a first polarizing filter 1002 (i.e. linearpolarizer) causing the nonpolarized light 1001 to become polarized light1003. Absent film 1005, the polarized light 1003 is filtered by thesecond polarizing filter 1004 and no, or little, light passes throughthe second polarizing filter 1004. By contrast, when polarized light 103passes through film 1005 the polarized light 1003 is rotated causingshifted polarized light 1006. This rotation allows some light to passthrough second polarizing filter 1004. Variables that change the amountof rotation or filtering of the light include the thickness of the film,stress of the film, strain of the film. The varying amount of rotationallows for the capture of an image as described above.

Referring now to FIG. 11, there is shown a ultraviolet vision system1100 according to one embodiment. Film 100 is transported into adarkened enclosure 1102. An ultraviolet light source 1101 illuminatesfilm 100 causing a detectable component in the film to excite and emitfluorescent electromagnetic energy. Ultraviolet image capture device1106 captures at least one image of the seal area of film 100. In anembodiment, a blue pass filter is used to filter out wavelengths oflight below 450 nm and about 525 nm. The blue pass filter improves usein higher than normal ambient lighting conditions. In one embodiment,either the light source or the blue pass filter transmits light ofbetween 425-495 nm.

In an embodiment, the ultraviolet camera is capable of capturing atleast two images of each seal as the film is in motion. In anembodiment, the ultraviolet camera is capable of capturing at least fourimages of each seal as the film is in motion. In an embodiment, theultraviolet camera is capable of capturing images of each seal at aspeed of at least 30 frames per second. In an embodiment, theultraviolet camera is capable of capturing images of each seal at aspeed of at least 60 frames per second. The properties of the seal allowfor the ultraviolet image capture device 1106 to capture an ultravioletimage of the seal along with variations in emitted energy along the sealarea. Such variations may be indicative of a defective seal. The imagedata captured by the ultraviolet image capture device 101 is sent to avision inspection engine 101 as described in more detail herein.

Example System Architecture

As illustrated in FIG. 12, in some embodiments, one or more computingsystems, such as a computing apparatus, may be used control the imagingdevice(s) 1204, light source(s) 1205, and vision system 1201. In someembodiments, one or more computing systems may control or direct othercomputing systems to control other functions within the packagingenvironment, such as in-line conveyor and materials handling devices.

The vision inspection engine 101 (such as shown in FIGS. 8, 9 and 11)may be a computing apparatus 1203 which may include a processor 1206, amemory 1207, input/output circuitry 1208, communications circuitry 1211,vision inspection circuitry 1210, and acquisition circuitry 1209, andmay be configured to execute the functions described herein. In someembodiments, the processor 1206 (and/or co-processor or any otherprocessing circuitry assisting or otherwise associated with theprocessor) may be in communication with the memory 1207 via a bus forpassing information among components of the apparatus. In someembodiments, the computing apparatus 1203 may be a distributed system ofcomputing components and/or a remotely located computing device. In someembodiments, the computing apparatus 1203 may be local or remote. Thememory 1207 may be nontransitory and may include, for example, one ormore volatile and/or non-volatile memories. In other words, for example,the memory may be an electronic storage device (e.g., a computerreadable storage medium). The memory may be configured to storeinformation, data, content, applications, instructions, or the like, forenabling the apparatus to carry out various functions in accordance withany example embodiment of the present invention.

The processor 1206 may be embodied in a number of different ways andmay, for example include one or more processing devices configured toperform independently. Additionally or alternatively, the processor mayinclude one or more processors configured in tandem via a bus to enableindependent execution of instructions, pipelining, and/ormultithreading.

In an example embodiment, the processor 1206 may be configured toexecute instructions stored in the memory 1207 or otherwise accessibleto the processor. Additionally or alternatively, the processor may beconfigured to execute hard-coded functionality. As such, whetherconfigured by hardware or software methods, or by a combination thereof,the processor may represent an entity (e.g., physically embodied incircuitry) capable of performing operations according to an embodimentof the present invention while configured accordingly. Alternatively, asanother example, when the processor is embodied as an executor ofsoftware instructions, the instructions may specifically configure theprocessor to perform the algorithms and/or operations described hereinwhen the instructions are executed.

In some embodiments, the vision system 1201 may include input/outputcircuitry 1208 that may, in turn, be in communication with processor1206 to provide output to the user and, in some embodiments, to receivean indication of a user input. The input/output circuitry 1208 maycomprise a user interface and may include a display and may comprise aweb user interface, a mobile application, a client device, a kiosk, orthe like. In some embodiments, the input/output circuitry 1208 may alsoinclude a keyboard, a mouse, a joystick, a touch screen, touch areas,soft keys, a microphone, a speaker, or other input/output mechanisms.The processor and/or user interface circuitry comprising the processormay be configured to control one or more functions of one or more userinterface elements through computer program instructions (e.g., softwareand/or firmware) stored on a memory accessible to the processor (e.g.,memory 1207, and/or the like).

Meanwhile, the communications circuitry 1211 may be any means such as adevice or circuitry embodied in either hardware or a combination ofhardware and software that is configured to receive and/or transmit datafrom/to a network and/or any other device or module in communicationwith the computing apparatus 1203. In this regard, the communicationcircuitry may include, for example, one or more cables (e.g., USB cable)connecting the imaging device (s) 1204 and light source(s) 1205 to thevision system 1201 for use with the software and hardware configurationsdescribed herein. In some embodiments, the communications circuitry 1211may include, for example, an antenna (or multiple antennas) andsupporting hardware and/or software for enabling communications with awireless communication network or one or more wireless devices.Additionally or alternatively, the communication interface may includethe circuitry for interacting with the cable(s) and/or antenna(s) tocause transmission of signals via the cable(s) and/or antenna(s) or tohandle receipt of signals received via the cable(s) and/or antenna(s).In some environments, the communication interface may additionally oralternatively support wired communication with a network (e.g.,Ethernet). As such, for example, the communication interface may includea communication modem and/or other hardware/software for supportingcommunication via cable, digital subscriber line (DSL), universal serialbus (USB), or other mechanisms.

The acquisition circuitry 1212 may be used to buffer the series ofimages and data captured at the imaging device(s) 1204 (e.g.,camera(s)). In some embodiments, the imaging device(s) 1204 may captureraw data incident one or more surfaces within the imaging device (e.g.,on one or more substrates of an image sensor). The imaging device(s)1204 may convert the raw data into computer-readable image data via oneor more circuitries, and may transmit the image data to the acquisitioncircuitry 1212. In some embodiments, image data may include any sensordata corresponding to a wavelength and/or intensity of electromagneticenergy used to detect defects in a packaged product. Image data mayinclude individual images; sequences of images; videos; and the like.The acquisition circuitry 1212 may further control the imaging device(s)1204 and light source(s) 1205 to trigger and time the respectiveillumination of the packaged product and capture of the raw dataaccording to any embodiments of the present invention. In someembodiments, the image data may be captured through any of the means forgenerating image data disclosed herein, which includes, but is notlimited to, any of the imaging devices (e.g., cameras, sensors, etc.)disclosed herein in both manual, autonomous, and partly-manual andpartly-autonomous forms of operation.

The vision inspection circuitry 1210 may be used to facilitateprocessing and analysis the image data received from the acquisitioncircuitry 1212. The vision inspection circuitry 1210 may further triggeran alert or other downstream notifications or actions based on theresult of the processing and analysis. The vision inspection circuitrymay further connect to one or more remote servers for data mining andstorage (e.g., to access models and data from which to train newmodels). In some embodiments, the vision system 1201 and visioninspection circuitry 1210 may comprise a single computing apparatus ormay comprise multiple apparatus connected locally or interacting over awired and/or wireless network.

In some embodiments, the vision system 1201 may include an operator userinterface (e.g., as part of input/output circuitry 1208. Defect data(e.g., from the vision inspection circuitry 1210) may be displayed onthe interface and archived either locally or remotely (e.g., via a localconnection or networked connection) in a database. Defect data andimages may be displayed real time on the interface. Instantaneous,historical, and statistical data may also be viewed on demand on theinterface in some embodiments. The computing apparatus 1203 can be setupto selectively detect and accurately classify defects in the packagedproduct, including detection of the excited fluorescence-based indicatorindicative of gas trapped within the packaging article.

Images of each defect can be classified, stored, displayed, and comparedwith prior and future images of other defects. The computing apparatus1203 may capture high-resolution images of each defect in real time.Discrete defect information such as individual defect geometricinformation and statistics of group defects can be provided forinstantaneous decision making and actions regarding process improvementand monitoring such as defect alarming. In some embodiments, eachpotential defect may be shown to an operator for a manual decision forhow to handle the defective packaged product. In some embodiments, thescreening, flagging, and/or separation of defective packaged productsmay be done partly or wholly automatically. Human operators may, in someinstances, verify the work of an otherwise automatic system.

Various outputs for marking/flagging, alarming, and autonomy can be setfor different defect severity levels. Data can be exported, for example,to MS Excel and/or a SQL database located anywhere on a network, withdata mining software allowing various reports to be easily generatedautomatically and/or on-demand. Defect data may be processed on aprocessing unit such as a digital processing board. Flagging can be usedin conjunction with separating and/or discarding packaged products withdamaged film or film with damage above a predetermined threshold.Flagging can be carried by applying a label to the film at (orcorresponding with) the location of the defect in the film for manual orautomatic separation (e.g., with a robotic package separator). In someembodiments, defective packages (e.g., packaged products showing a leakor low-vac condition) may be unpackaged and repackaged with a newpackaging article.

In an embodiment, the input/output circuitry 1208 may allow for externalsignal inputs such as new roll or new production run indication andpause inspection indication. Outputs for alarms on user-defined defectalarm criteria are also handled through the input/output circuitry 1208(e.g., the user interface). Outputs can also be initiated to controldownstream flagging or marking devices. Alarms can be activated fordefects of different pre-defined seventies or criteria. Alarm and defectinformation from the computing apparatus 1203 can be sent via OPC (i.e.,software interface standard) to the plant network, programmable logiccontroller (PLC), or supervisory control and data acquisition/humanmachine interface (SCADA/HMI).

In an embodiment, an encoder (not shown) may be used to measure conveyorspeed so that the location of a detected defective packaged product isascertainable. A series of pulses from the encoder is received by thesystem and counted. The count is sent to the processor 1206 to determinethe distance down the conveyor at which the detected defective packagedproduct is located, and may be used to time operation of a defectseparator to remove the defective packaged product from the line priorto pack off (when the packaged product moves off of the productionline).

While a local system is depicted, it is understood that a distributedsystem may be utilized by connecting a plurality of computing apparatusvia a network. In some embodiments, the network may include any wired orwireless communication network including, for example, a wired orwireless local area network (LAN), personal area network (PAN),metropolitan area network (MAN), wide area network (WAN), or the like,as well as any hardware, software and/or firmware required to implementit (such as, e.g., network routers, etc.). For example, network mayinclude a cellular telephone, an 802.11, 802.16, 802.20, and/or WiMaxnetwork. Further, the network may include a public network, such as theInternet, a private network, such as an intranet, or combinationsthereof, and may utilize a variety of networking protocols now availableor later developed including, but not limited to TCP/IP based networkingprotocols.

In some embodiments, the plurality of computing apparatuses maycollectively include the features of the computing apparatus 1203described above with respect FIG. 4. In some embodiments, any number ofcomputing apparatuses may make up the computing apparatus.

Example Detection Models and Algorithms

In some embodiments, various algorithms, models, and processes may beimplemented to detect defects in the seal of product packaging andpackaged product. Each of the algorithms, models, and processes may beconfigured to locate and identify seal defects. In some embodiments, anartificial intelligent algorithm may be implemented (e.g., as shown inFIG. 13), while in other embodiments, an algorithmic approach may beused based on received sensor information (e.g., as shown in FIG. 14).The processes disclosed herein (e.g., algorithmic solutions, model-basedsolutions, etc.) may receive image data in one or more states ofprocessing and pre-processing; may further process, sort, classify, andanalyze the image data; and may identify seal defects present in thepackaging article based thereon.

FIG. 13 illustrates a flow diagram of a model-based process 500(depicted as a functional block diagram) for assessing whether a seal ofa packaging article has a defect. The process 500 may be embodied in andexecuted by, for example, the vision systems shown in FIGS. 8, 9 and 11herein. As depicted, the system may include one or more imaging devices210, a data acquisition system 502 (e.g., utilizing the computingapparatus 1203 and acquisition circuitry 1212 shown in FIG. 12), avision inspection engine 504 (e.g., utilizing the computing apparatus1203 shown in FIG. 12 and vision inspection circuitry 1210 shown in FIG.12), an in-line action system 506, a database management system 508, adata collecting system 510 (e.g., utilizing the computing apparatus 1203shown in FIG. 12 and data collection circuitry 1213), and a modelgenerator 520 (e.g., utilizing the computing apparatus 1203 shown inFIG. 12 and process generation circuitry 1209 shown in FIG. 12). FIG. 13shows an example flow diagram for executing a process 500 describedaccording to some embodiments.

In the depicted embodiment, using an appropriate combination of animaging device (e.g., including lens and/or sensor choice) and lighting(e.g., via light source(s) 215), a series of images are acquired and fedinto the acquisition system 502 where the data is acquired, buffered,and transferred to one of the vision inspection engine 504 or datacollecting system 510. The depicted embodiment of FIG. 13 includes twoexample use cases, a model generation workflow and a model applicationworkflow.

In a model generation workflow, the process 500 may generate a modelfrom a plurality of images. Images may be captured by the imagingdevice(s) 210 and received by the acquisition system 502. Theacquisition system may include an image acquisition engine 540,acquisition routine 542, memory buffer 544, and workspace 546. Theacquisition system 502 may buffer the image and transmit the image tothe data collecting system 510, which may label 512 the images (e.g.,good/sufficient seal or bad/weak seal) and store the images 514. In someembodiments, the model generation workflow may retrieve images directlyfrom storage 514 or the images (e.g., a training set) may be loaded intothe system separately (e.g., via communications circuitry 1211 shown inFIG. 12).

The images may be labeled 512 using any process described herein. Forexample, a user may input a label into the system in association witheach image. In some embodiments, the packaging article may include a tagor label that identifies the characteristics of the packaging article tothe user or to a vision system for association with the respectiveimages to the packaging article.

From the data collecting system 510, the labeled images may be input tothe model generator 520 to generate one or more models, which mayreceive the images in a first step 521 by receiving the labeled imagesin an image database 522 and initializing the images 523 for use in themodel generator. The initialized images may then be passed into imageanalysis 524 where the images may be pre-processed 525 and may beanalyzed 526. The model generation 527 may then be performed to createmultiple models 528 based on the analyzed images using the trainingprocess described herein, and the models may be tested and a preferredmodel selected 529 in some embodiments based on the model evaluationmatrix (which includes parameters such as accuracy, false positives,true positives, recall, precision, and others) of its predictions in thetest data.

Once the model is created and selected in the model generator 520, themodel may be deployed to the vision inspection engine 504 in a modelapplication workflow. In the model application workflow, an image may becaptured by the imaging device(s) 210 and fed into the acquisitionsystem 502 where the data is acquired, buffered, and transferred to thevision inspection engine 504. In the vision inspection engine 504, themodel may be applied to an unknown image to classify the image (e.g.,good/sufficient seal or bad/weak seal) based on the artificialintelligent algorithm developed by the model generator 520 in the modelgeneration workflow. The process may include initializing the model 532,the image may be input into the decision function 534 to receive adecision at the detection model output 536.

The detection results 538 may be fed into the in-line action system 506to set up predetermined alarms, film flagging, displaying an image of afaulty seal (e.g., via the user interface), displaying data pertainingto one or more defects including displaying data related to geometriccharacteristics of the faulty seal area, location of the defect, degreeof occurrence of defects; severity of defects, generating a report ofdefect data and/or any other desired output. Data pertaining to defectscan be displayed instantaneously and online, or after production iscomplete (i.e., offline or not-on-the-fly), the data being accessible inan offline database management system 508. Using data mining, the datacan be manipulated, visualized, and organized into any on-demand reportforms desired.

The detection results may further be transferred from the visioninspection engine 504 to the data collecting system 510 as labeledimages to be stored in image storage 514 and used in subsequent modelgeneration processes to recursively reduce loss of the objectivefunction and improve the models' performance based on model evaluationmatrix.

The data processing software and hardware may be set up to accommodatedifferent seal types with minimum need for on-the-fly adjustment ofparameters such as exposure time and light intensity. In someembodiments, portions of the process 500 may be done offline orremotely. For example, the data collecting system 510 and/or modelgenerator 520 may be remotely located from the remainder of the process(e.g., performed on a server). Moreover, some of or all of the visioninspection process may occur remotely.

Turning to FIG. 14, an example flow diagram of an algorithmic process550 is shown (depicted as a functional block diagram) for assessingwhether a packaging article has a faulty seal. The process 550 may beembodied in and executed by, for example, the vision systems shown inFIGS. 8, 9 and 11 herein. As depicted, the system may include one ormore imaging devices 210, a data acquisition system 502 (e.g., utilizingthe computing apparatus 1203 and acquisition circuitry 1212 shown inFIG. 12), a vision inspection engine 504 (e.g., utilizing the computingapparatus 1203 shown in FIG. 12 and vision inspection circuitry 1210shown in FIG. 12), an in-line action system 506, a database managementsystem 508, a data collecting system 510 (e.g., utilizing the computingapparatus 1203 shown in FIG. 12 and data collection circuitry 1213), andan inference engine 560 (e.g., utilizing the computing apparatus 1203shown in FIG. 12 and process generation circuitry 1209 shown in FIG.12). FIG. 14 shows an example flow diagram for executing a process 550described according to some embodiments.

In the depicted embodiment, using an appropriate combination of animaging device(s) 210 (e.g., including lens and/or sensor choice) andlight source(s) 215 (e.g., via illuminators, UV lighting, polarizedlight), a series of images are acquired and fed into the acquisitionsystem 502 where the data is acquired, buffered, and transferred to oneof the vision inspection engine 504 or data collecting system 510. Thedepicted embodiment of FIG. 14 includes two example use cases, analgorithm generation workflow and an algorithm application workflow.

In an algorithm generation workflow, the process 550 may generate analgorithm from a plurality of images. Images may be captured by theimaging device(s) 210 and received by the acquisition system 502. Theacquisition system may include an image acquisition engine 540,acquisition routine 542, memory buffer 544, and workspace 546. Theacquisition system 502 may buffer the image and transmit the image tothe data collecting system 510, which may label 512 the images (e.g.,good/sufficient seal or bad/weak seal) and store the images 514. In someembodiments, the algorithm generation workflow may retrieve imagesdirectly from storage 514 or the images (e.g., a test set of images) maybe loaded into the system separately (e.g., via communications circuitry1211 shown in FIG. 12).

The images may be labeled 512 using any process described herein. Forexample, a user may input a label into the system in association witheach image. In some embodiments, the packaged product may include a tagor label that identifies the characteristics of the packaged product tothe user or to a vision system for association with the respectiveimages of the product. In some embodiments, a training set may bepre-loaded with labels.

From the data collecting system 510, the labeled images may be input tothe inference engine 560 to generate and identify one or more algorithmsthat may receive the images in a first step 561 by receiving the labeledimages in an image database 562 and initializing the images 563 for usein the inference engine. The initialized images may then be passed intoimage analysis 564 where the images may be pre-processed 565 and may beanalyzed 566. The algorithm determination 567 may then be performed tocreate one or more algorithms 568 based on the analyzed images using theprocess described herein, and the algorithms may be tested and apreferred algorithm selected 569 in some embodiments based on the modelevaluation matrix of its predictions in the test data. In someembodiments, one hundred test images are used. In some embodiments, onethousand test images are used. In some embodiments, a user may manuallyidentify desired features in the images and/or image processingparameters to identify the features during or before algorithmgeneration. In some embodiments, the system may partly or fullyautonomously process the images and/or detect desired features in theimage using the techniques described herein.

Once the algorithm is created and selected in the inference engine 560,the algorithm may be deployed to the vision inspection engine 504 in analgorithm application workflow. In the algorithm application workflow,an image may be captured by the imaging device(s) 210 and fed into theacquisition system 502 where the data is acquired, buffered, andtransferred to the vision inspection engine 504. In the visioninspection engine 504, the algorithm may be applied to the unknown(e.g., unlabeled) image to classify the image (e.g., good/sufficientseal or bad/weak seal) based on the algorithm selected in the inferenceengine 560 in the algorithm generation workflow. The process may includeinitializing the algorithm 572, the image may be input into the decisionfunction 574 to receive a decision at the detection algorithm output576.

The detection results 578 may be fed into the in-line action system 506to set up predetermined alarms, film flagging, displaying an image theseal area (e.g., via the user interface), displaying data pertaining toone or more defects including displaying data related to geometriccharacteristics of the seal, location of the defect, degree ofoccurrence of defects; severity of defects, generating a report ofdefect data and/or any other desired output. Data pertaining to defectscan be displayed instantaneously and online, or after production iscomplete (i.e., offline or not-on-the-fly), the data being accessible inan offline database management system 508. Using data mining, the datacan be manipulated, visualized, and organized into any on-demand reportforms desired.

The detection results may further be transferred from the visioninspection engine 504 to the data collecting system 510 as labeledimages to be stored in image storage 514 and used in subsequentalgorithm generation processes to improve the algorithms' evaluationmatrix. In some embodiments, the image analysis and algorithm generationin the inference engine 560 may be performed autonomously. In someembodiments, the image analysis and algorithm generation in theinference engine 560 may be performed partly manually. In someembodiments, the image analysis and algorithm generation in theinference engine 560 may be performed manually.

The data processing software and hardware may be set up to accommodatedifferent concentration levels with minimum need for on-the-flyadjustment of parameters such as exposure time and light intensity. Insome embodiments, portions of the process 550 may be done offline orremotely. For example, the data collecting system 510 and/or inferenceengine 560 may be remotely located from the remainder of the process(e.g., performed on a server). Moreover, some or all of the visioninspection process may occur remotely.

In some example embodiments, an algorithmic process may be used todetect seal quality of the packaging article. For example, in someembodiments, a feature-extraction based algorithm may be used to detectportions of the seal in the packaging article exhibiting excitedfluorescence above a predetermined threshold intensity or at apredetermined wavelength or change in wavelength from incident light. Insome embodiments, a feature-extraction based algorithm may be used todetect portions of the seal in the packaging article exhibiting thermalgradients distinct from baseline thermal gradients. In some embodiments,dark feature detection may be used to identify dark spots and lightspots in the captured image data, which may separate areas of lowfluorescence from areas of high fluorescence, with the high fluorescenceareas indicating a likelihood of a faulty seal. In some embodiments,light feature detection may be used in a similar manner to dark featuredetection. In some embodiments, dark feature detection and light featuredetection may be calculated based on grey scale value thresholds of theimage data. In some embodiments, feature extraction may includeextracting one or more textures of the image. The image textures may becompared to reference images to identify a faulty seal. In someembodiments, feature extraction may further be used to exclude undesiredareas such as background or an area around the seal. For example, insome embodiments, the seal of the packaging article may have aninconsistent texture or thickness, which may produce a false positive onthe captured images. In some embodiments, the bright spot caused by theseal may be excluded from the algorithm.

In some embodiments, the algorithmic process may comprise streakdetection. The streak detection may be based on extracted geometryfeatures in the captured images.

With reference to FIG. 15, in an example embodiment, the algorithm 600(e.g., an algorithm generated by the inference engine 560) andassociated image processing may comprise the following steps: (1)capture an image 602; (2) apply de-noising to the image 604; (3) convertthe image to greyscale 606; (4) apply thresholding to reduce the sectionof the image for consideration 608; (5) in some embodiments, eliminateundesired areas of the image 610; and (6) identify features in theremaining image corresponding to a faulty seal 612. In some embodiments,the algorithm 600 may further include classifying an image based on thedetected features 614. In some embodiments, steps of the aforementionedalgorithm may be eliminated so long as the faulty seal can beidentified.

As noted above, in some embodiments, an artificial intelligent algorithm(e.g., detection model shown in FIG. 13) may be developed to classifyimage data of the seal (e.g., trained using the model generator 520shown in FIG. 13). Depicted in FIG. 16 is an embodiment of a method 700of developing a trained image classification model. At block 702,training image data of seal areas is obtained. In some embodiments, thetraining image data includes images and/or video (i.e., a sequence ofimages) of seals having a known state. In some embodiments, the visionsystem used to obtain the training image data is the same as the visionsystem that will be used to obtain image data of seal area of thepackaging article of an unknown state after the trained imageclassification model is created (e.g., the vision system in the finalproduction line). In some embodiments, a test bed or other experimentalconfiguration may be used to obtain the training image data. At block704, the training image data is manually labeled with the states of theseal area of the packaging article in the training image data. Forexample, a user can manually input a state (e.g., the packaging articlehas a defective seal, has an adequate seal, or has a particular defect)for each image and/or video of a seal in the image data. Manuallylabeling the image data may include physically testing the seals todetermine whether individual seals are adequate or defective and thenapplying a label to the image data based on the results of the physicaltesting. The training samples may include a plurality of seals and aplurality of defective seals. In some embodiments, the number of sealsrepresented in the training image data is in a range of tens of seals,hundreds of seals, thousands of seals, or more. At these numbers, themanual labeling process of the training image data may be a labor- andtime-intensive process. At block 706, the labeled training image date isinput into a training module.

In some embodiments, the training model is a machine learning module,such as a “deep learning” module. Deep learning is a subset of machinelearning that generates models based on training data sets provided toit. In some embodiments, the training model may use supervised learningtechniques, semi-supervised learning techniques unsupervised learningtechniques including clustering, anomaly detection, Hebbian Learning, aswell as learning latent variable models such as Expectation-maximizationalgorithm, method of moments (mean, covariance), and Blind signalseparation techniques, which include principal component analysis,independent component analysis, non-negative matrix factorization, andsingular value decomposition. In one example, unsupervised learningtechniques were utilized in combination with several imagepre-/post-processing techniques including imaging texture analysis(e.g., calculating Leaker Detection Index values) to achieve detectionand classification. In another embodiment, the unsupervised learningtechnique Anomaly Detection was utilized in combination with severalimage pre-/post-processing techniques including imaging texture analysisto achieve detection and classification.

In embodiments, unsupervised learning techniques include the followingsteps. 1) Data Collection where images are collected for all types ofseals without any category definition. Best practices are implemented toensure only baseline seals are produced. 2) Algorithm Development whereautoencoder components like encoder, decoder and latent space vector aredesign based on the input data specification and expected output. 3)Algorithm Training where the autoencoder neural network is trained onthe baseline seals dataset with object to reproduce the samples withfinest details. 4) Defect Segregation where post training theautoencoder is deployed into production. The input seals that produceerror divergent from the baseline examples are segregated. 5) AlgorithmRe-Training where segregated defects are validated by a human. A set offalse positive samples is prepared and used to retrain the autoencoderto further improve the accuracy. The process may be repeated to improveperformance.

In embodiments, semi-supervised learning techniques include thefollowing steps. 1) Segregate defective seals from baseline usingautoencoders. 2) Clustering defective seals by grouping defective sealsbased on a similarity score using clustering algorithm. 3) Manualcategorization by a human evaluating defective seal cluster andassignment of categories. 4) Automatic categorization which requiresdeveloping a classification algorithm based on the manually categorizeddefects.

At block 708, the artificial intelligent algorithm is developed toclassify seals. In some embodiments, as the artificial intelligentalgorithm is developed, one or more learning algorithms are used tocreate the artificial intelligent algorithm based on the labeled statesof the seals in the training image data. In some embodiments, theartificial intelligent algorithm is created based on one or more inputvectors which are indicative of a characteristic of the seal. In oneexample, the input vector may be pleats in the seal. In another example,the input vectors may be the properties of the fluorescentelectromagnetic energy emitted by excited fluorescence-based indicatorin the film in accordance with the particular indicator and illuminationused, as described above. For example, the input vectors may include oneor more of a wavelength of the fluorescent electromagnetic energyemitted by an excited fluorescence based indicator; an intensity of thefluorescent electromagnetic energy emitted by the excitedfluorescence-based indicator; and/or a change in wavelength of thefluorescent electromagnetic energy emitted by the excitedfluorescence-based indicator compared to the incident electromagneticenergy from the illuminators. In some embodiment, the input vectors maycorrespond to a wavelength of the fluorescent electromagnetic energyemitted by the excited fluorescence-based indicator being one or morecolors in the visible spectrum, detection of an additive in a film ofthe packaging article using a non-visible electromagnetic energy (e.g.,ultraviolet, infrared), thermal imaging scans, visible light scans,photoelasticity scans, the presence and numbers of film folds, or anyother number of possible input vectors. In an embodiment the lightsource(s) 215 can be an ultraviolet backlight with software forcontrolling shutter speed and light intensity. In other embodiments, thelight source(s) 215 are white light. In some embodiments, the wavelengthof the fluorescent electromagnetic energy emitted by the excitedfluorescence based indicator may be at least in the ultraviolet range.In some embodiments, the wavelength of the fluorescent electromagneticenergy emitted by the excited fluorescence based indicator may be atleast in the visible spectrum. In some embodiments, the wavelength ofthe fluorescent electromagnetic energy emitted by the excitedfluorescence based indicator may be at least in blue or violet range. Insome embodiments, the wavelength of the incident light from theilluminators may be at least in the ultraviolet range. In someembodiments, the wavelength of the incident light from the illuminatorsmay define a peak in the ultraviolet range. In some embodiments, thewavelength of the incident light may be in the ultraviolet range and thefluorescent electromagnetic energy emitted by the excitedfluorescence-based indicator may be in the visible range. In embodimentsin which the process is designed to simultaneously inspect multiple filmlayers at the same time for the same film, multiple light sources 215and/or multiple imaging devices 210 can be used with one or morecontrols for shutter speed, light intensity, light source, backlightingor over lighting.

The use of input vectors for training may help the artificialintelligent algorithm identify defective seal without identifying theunderlying cause. For example, a seal may have a small pleat that wouldbe difficult to detect using image data captured as the packagingarticle is being moved on a transportation system. The use of the inputvectors, fluorescence-based indicator, thermal image data,photoelasticity image data, detailed herein allows the artificialintelligent algorithm to detect that the seal is defective without theneed to identify the defect itself. After the input vectors are modeled,an artificial intelligent algorithm can be developed as adecision-making process based on a number of the input vectors. Examplesof decision-making processes include decision trees, neural networks,and the like. In some embodiments, the decision-making process of theartificial intelligent algorithm is based on a determination of anacceptable arrangement of the input vectors in the decision-makingprocess.

The result of the development of the artificial intelligent algorithm inblock 708 is the artificial intelligent algorithm depicted at block 710.The artificial intelligent algorithm can be used during normal operation(e.g., operation that is not used to train to the artificial intelligentalgorithm) to identify states of seal. In some embodiments, theartificial intelligent algorithm includes a neural network that has anumber of layers. Depicted in FIG. 20 is an example of a neural network1800 that is a multilayer neural network. In the depicted embodiment,the neural network 1800 includes a first layer 1802 with three inputnodes, a second layer 1804 with five hidden nodes, a third layer 1806with four hidden nodes, a fourth layer 1808 with four hidden nodes, anda fifth layer 1810 with one output node.

The neural network 1800 also includes a first set of connections 1812between each pair of the three input nodes in the first layer 1802 andthe five input nodes in the second layer 1804, a second set ofconnections 1814 between each pair of the five input nodes in the secondlayer 1804 and the four hidden nodes in the third layer 1806, a thirdset of connections 1816 between each pair of the four hidden nodes inthe third layer 1806 and the four hidden nodes in the fourth layer 1808,and a fourth set of connections 1818 between each pair of the fourhidden nodes in fourth layer 808 and the output node in the fifth layer1810. In some embodiments, the input nodes represent inputs into theartificial intelligent algorithm (e.g., image data, metadata associatedwith the image data, etc.), one or more of the hidden nodes (e.g., oneof the layers of hidden nodes) may represent one of the input vectorsdetermined during the development of the model, and the output noderepresents the determined state of the seal.

Referring now to FIG. 17 is an embodiment of a method 900 of using atrained image classification model to classify a state (e.g., pleat,weak seal, cold seal, channel pleat) of a seal. At block 902, image dataof the seal is acquired (e.g., by imaging device(s) 210). The image dataof the seal may be obtained by a vision system. In some embodiments, theimage data of the seal is obtained while the packaging article is beingtransported by a transport system. It is understood that the image datacan be captured after forming seals in the formation of the packagingarticle, or after forming of seals to seal a product in the packagingarticle.

At block 904, the image data of the seal is input into a trained imageclassification model. The trained image classification model may beoperating on a computing device or a remote computing device from thelocal computing device. The trained image classification model isconfigured to classify a state of the seal based on the image data. Atblock 906, a classification of a state of the seal is received from thetrained image classification model. In some embodiments, the classifiedstate includes an indication that the seal is defective, isnon-defective, or has a particular defect, and/or an indication of adegree of certainty as to whether the seal is defective, isnon-defective, or has a particular defect. In some embodiments, theclassified state is received by one or more of displaying theclassification on a user interface output device, communicating theclassification via a communication interface to one or more externaldevices, or storing the classification in a database. In someembodiments, the received classification the seal includes one or moreof the classified state or the seal or a degree of certainty of theclassified state of the classified state of the seal. In one specificexample, the state is communicated to a routing system (e.g., a pack-offapparatus) that is configured to route the packaging article on atransportation system based on their seal states, such as routingdefective packages to a location for testing, repackaging, recyclingand/or waste disposal.

As noted above, the method 700 is used to obtain the trainedclassification model at block 710 and then the trained classificationmodel can be used in method 900 to classify seals. In some embodiments,the training image data acquired at block 702 is image data of a sealfor a particular type of product packed in a packaging article and theimage data acquired at block 902 is image data of the same type of sealfor the particular type of product packed in a packaging article. In oneexample, the training image data acquired at block 702 is image data ofthe seal and the image data acquired at block 902 is image data of thesame type of seal. In some embodiments, the training image data acquiredat block 702 is image data of a particular type seal and the image dataacquired at block 902 is image data of a different type of seal. Eventhough the seal imaged in-line may be a different type from the sealused in the training set, the trained classification model using thetraining image data from the training seal image data may be able toclassify states of the seal with sufficient accuracy.

Depicted in FIG. 18 is an embodiment of a method 1010 of developing atrained image classification model. At block 1012, training image datais acquired for a number of seals. At block 1014, the training imagedata is manually labeled as defective or non-defective. The manuallabeling of the training image data may be done by a user entering anindication of defective or non-defective for each of the sealsrepresented in the training image data into a user interface inputdevice of a computing apparatus. In some embodiments, training packagingarticles having seals may be labeled with their respective status (e.g.,defective or non-defective). Training packaging articles are packagingarticles with either known defective seals, or known non-defectiveseals. The training packaging articles may further identify particulardefects. The training packaging articles are used to train the model toimproves reliability of the model.

At block 1016, model information, training objectives, and constraintsare initialized. In some examples, model information includes a type ofmodel to be used, such as a neural network, a number of input vectors,and the like. In some examples, training objectives can include adesired or expected performance of the artificial intelligent algorithm,such as an evaluation matrix has a confidence rating of greater than orequal to a predetermined rate (e.g., greater than or equal to one ormore of 90%, 95%, 96%, 97%, 98%, or 99%). In some examples, constraintscan include limitations of the artificial intelligent algorithm, such asa minimum number of layers of a neural network, a maximum number oflayers of a neural network, a minimum weighting of input vectors, amaximum weighting of input vectors, or any other constraints of anartificial intelligent algorithm. At block 1018, the model can betrained using the model information and the model constraints. In someembodiments, the training image data is separated into two subsets—atraining subset and a validation subset—and the training of the model atblock 1018 includes training the model using the training subset of theimage data.

At block 1020, a determination is made whether the training objective ismet. In some embodiments, the determination at block 1020 is made bycomparing the results of the artificial intelligent algorithm to thetraining objective initialized at block 1016. In some embodiments, wherethe training image data is separated into the training subset and thevalidation subset, the determination at block 1020 includes testing themodel trained at block 1018 using the validation subset of the imagedata. If, at block 1020, a determination is made that the trainingobjective is not met, then the method 1010 proceeds to block 1022 wherethe training objective and/or the constraints are updated. After thetraining objective and/or the constraints are updated at block 1022, themethod 1010 returns to block 1018 where the model is trained using theupdated training objective and/or constraints. If, at block 1020, adetermination is made that the training objective is met, then themethod 1010 proceeds to block 1024 where the artificial intelligentalgorithm is stored. Storing the artificial intelligent algorithm mayinclude storing the artificial intelligent algorithm in one or morememories in a computing device (e.g., a local computing device, a remotecomputing device, etc.).

In some embodiments, a vision system may be used both to train a modelto classify states of seal and to apply the artificial intelligentalgorithm to classify states of seal. Depicted in FIG. 19 is anembodiment of a method 1100 for a vision inspection engine to both traina model to classify states of seals for packaging articles and apply theartificial intelligent algorithm to classify states of seals. In someembodiments, the vision system includes an image sensor system and acomputing apparatus which may define a vision inspection engine 504 andan acquisition system 502. In those embodiments, the model may operateon the computing apparatus while the imaging device(s) obtains imagedata of the seal either for training or applying the model.

At block 1102, the vision system and the classification model areinitialized. In some embodiments, initialization of the vision systemincludes initializing a computing apparatus and initializing imagingdevice(s), and initialization of the classification model includesloading launching software that includes the classification model on thecomputing apparatus. At block 1104, the image data of a seal is acquired(e.g., by imaging device(s) and acquisition system). In someembodiments, the imaging device(s) acquires the image data of the sealand provides the image data to the computing apparatus. At block 1106, adetermination is made whether the classification model is in trainingmode. The determination may be made by the software operating on thecomputing system that includes the classification model.

If, at block 1106, a determination is made that the classification modelis in training mode, then the model passes to block 1108, where adetermination is made if a state is available for the seal. A state maybe available for a seal when a user manually enters a state for the sealinto a computing device or scans a state of the seal (e.g., from a barcode or other indicia on the training packaging article). If, at block1108, a determination is made that a state is available, then the methodproceeds to block 1110. At block 1110, the classification model isupdated based on the image data and the state for the seal. Updating theclassification model can include any of the methods described herein fortraining and/or developing classification models. The seal state (e.g.,the manually-entered state) is available, as shown in block 1112.However, if, at block 1106, a determination is made that theclassification model is not in training mode or if, at block 1108, adetermination is made that a state is not available, then the methodproceeds to block 1114.

At block 1114, the classification model classifies a state of the seal.In some embodiments, the state of a seal classified by theclassification model includes a determination of whether the seal isdefective (e.g., pleat or weak seal), is non-defective, or has aparticular defect, and an indication of a degree of certainty as towhether the seal is defective, is non-defective, or has a particulardefect. At block 1116, a determination is made whether a confidencelevel of the classified state is low or high. In some embodiments, theconfidence level is a percentage representing the degree of certaintythat the classified state of the seal is accurate and confidence levelis low if the degree of certainty is below a predetermined percentage ofan acceptable degree of certainty. For example, if the acceptable degreeof certainty is 90%, then the classified state of the seal is deemed tobe low if the degree of certainty of the classified state is below 90%.If, at block 1116, the confidence level is determined to not be low,then the seal state has been determined, as shown at block 1118.However, if at block 1116, the confidence level is determined to be low,then the method proceeds to block 1120 where the seal product is setaside for off-line and/or manual classification (e.g., classification bya user after visual inspection or physical testing separate from theproduction line). In embodiments where multiple vision systems areutilized, either redundant or different types (e.g. thermal andphotoelasticity), the likelihood of increasing the confidence isimproved.

If a state of the seal is available, either at block 1112 or at block1118, then the method proceeds to block 1122. At block 1122, the stateof the seal is output. In some embodiments, outputting the state of theseal includes one or more of displaying the state of the seal on a userinterface output device, communicating the state of the seal via acommunication interface to one or more external devices, or storing thestate of the seal in a database. In some embodiments, the state of theseal includes one or more of an indication of whether the seal isdefective, is non-defective, or has a particular defect, or a degree ofcertainty of whether the seal is defective, is non-defective, or has aparticular defect.

Whether state of the seal is output at block 1122 or the packagingarticle is held for manual classification at block 1120, the method 1100then proceeds to block 1124. At block 1124, a determination is madewhether another packaging article is available. In some embodiments, thedetermination at block 1124 can be based on whether another packagingarticle is detected on the conveyor (e.g., via one or more sensors). Insome embodiments, the determination at block 1124 can be based onwhether a user inputs an indication whether another packaging article isavailable. If, at block 1124, a determination is made that anotherpackaging article is not available, then, at block 1126, the visionsystem and the classification model are shut down. However, if, at block1124, a determination is made that another packaging article isavailable, then the method 1100 loops back to block 1104 where imagedata is acquired of the seal of the next packaging article and themethod 1100 proceeds from block 1104 as described above for the nextpackaging article.

As discussed above, an artificial intelligent algorithm to classifystates of seals for packaging articles from image data may include onedecision-making process, such as a decision tree or a neural network. Insome embodiments, an artificial intelligent algorithm to classify statesof seal from image data may include more than one decision-makingprocess. Depicted in FIG. 21 is an embodiment of a method 1200 ofclassifying a state of a seal. In the depicted embodiment, the method1200 is performed in part by an image sensor system 1202, a detectiondecision-making process 1204, a classification decision-making process1206, and an output device 1208. At block 1210, the image sensor systemacquires image data of a seal. In some embodiments, the image sensorsystem 1202 may acquire the image data as the packaging article is beingtransported by a transport system. After the image data is acquired atblock 1210, the image sensor system has image data 1212 that can becommunicated to the detection decision-making process 1204. In oneembodiments, the detection decision-making process 1204 is asoftware-based decision-making process operating on one or morecomputing devices.

At block 1214, the detection decision-making process 1204 processes theimage data received from the image sensor system 1202. In someembodiments, the processing of the image data at block 1214 is performedby an artificial intelligent algorithm that has been trained to detect aregion of interest associated with a seal of a packaging article inimage data. In some embodiments, the processing of the image data atblock 1214 includes one or more of cropping an image in the image dataaround a detected seal or seals in the image, selecting a frame or asubset of frames from a video in the image data, identifying irrelevantpixels from an image in the image data and replacing the irrelevantpixels with the least significant values of the image data. In someembodiments, the processing of the image data produces a single imagehaving a rectangular shape with the identified seal substantiallycentered in the image and the pixels deemed to be irrelevant beingreplaced with the least significant values. In some embodiments, theprocessing of the image data can include masking a portion of an image,where areas of the image outside of a region of interest (e.g., outsideof the seal) are replaced with low value data (e.g., the pixels are allchanged to black) to reduce the amount of processing to classify thestate of the seal and reduce the likelihood of error when classifyingthe state of the seal.

In one embodiment of processing image data, a custom boundary isconstructed around a representation of a seal in the image data. Abounding box encompassing the seal is also constructed in the customboundary. The processing also includes cropping the bounding box fromthe entire image data. One advantage of cropping the image data based onthe custom boundary is that the later classification of the state of theseal may be limited to areas of interest without the need to inspectareas of the image data that are not of interest. This may, in turn,increase the confidence level of classification and therefore overallperformance of the classification. In some embodiments, where thedetection decision-making process 1204 is a multilayer neural network,creating the bounding box around the custom boundary simplifiescompatibility requirements between the image data and the first layer ofthe neural network. Additionally, cropping the image data results in aportion of the image data being processed for classification, ratherthan the entire image data, which reduces the processing time forclassification. In some embodiments, the custom boundary may help ingenerating a numerical value for one or more of the area of the seal,its centroid, or its orientation.

At block 1216, a determination is made whether the presence of a seal isdetected in the image data. In some embodiments, the determination madeat block 1216 is a part of the processing of the image data at block1216. In some embodiments, the determination of whether the seal isdetected at block 1216 is a separate process from the processing of theimage data at block 1216. If, at block 1216, a determination is madethat the presence of seal is not detected, then the method 1200 proceedsto block 1218 where the image data is discarded (e.g., deleted) and themethod 1200 ends. However, if, at block 1216, a determination is madethat the presence of a seal is detected, then the processed image datarepresented at block 1220 can be communicated to the classificationdecision-making process 1206. In some embodiments, the classificationdecision-making process 1206 is a software-based decision-making processoperating on one or more computing devices, which may be the same as ordifferent from the one or more computing devices on which the detectiondecision-making process 1204 operates. In some embodiments, processingthe image data at block 1214 to obtain the processed image data, asshown at block 1220, prior to classifying a state of the sealrepresented in the data increases the evaluation matrix of thelater-performed classification by the classification decision-makingprocess 1206.

At block 1222, the classification decision-making process 1206classifies the processed image data received from the detectiondecision-making process 1204. In some embodiments, the classification ofthe image data at block 1222 is performed by an artificial intelligentalgorithm that has been trained to classify a state of a sealrepresented in processed image data. In some embodiments, theclassification of the state of the seal represented in the processedimage data at block 1222 includes a determination of whether the seal isdefective, is non-defective, or has a particular defect. In someembodiments, the classification of the state of the seal represented inthe processed image data at block 1222 includes a determination ofwhether the seal is defective, is non-defective, or has a particulardefect, and an indication of a degree of certainty as to whether theseal is defective, is non-defective, or has a particular defect.

At block 1224, a determination is made whether a confidence level of theclassified state is low. In some embodiments, the confidence level is apercentage representing the degree of certainty that the classifiedstate of the seal is accurate and the confidence level is low if thedegree of certainty is below a predetermined percentage of an acceptabledegree of certainty. For example, if the acceptable degree of certaintyis 90%, then the classified state of the seal is deemed to be low if thedegree of certainty of the classified state is below 90%. If, at block1224, the confidence level is determined to not be low, then the sealstate has been determined, as shown at block 1226. However, if at block1224, the confidence level is determined to be low, then the methodproceeds to block 1228 where the seal and/or the image data is flaggedfor manual classification.

At block 1230, a state of the seal is manually classified outside of theclassification decision-making process. In some embodiments, the seal ismanually classified by a user after visual inspection or physicaltesting of the packaging article. At block 1232, the user inputs themanually-classified state of the seal to the classificationdecision-making process 1206. At block 1234, the classificationdecision-making process 1206 is updated. In embodiments where theclassification decision-making process 1206 is an artificial intelligentalgorithm, updating the classification decision-making process 1206includes further training the artificial intelligent algorithm based onthe manual classification. After updating the classificationdecision-making process 1206, the method 1100 returns to block 1226where the classified state of the vacuum-packaged product is themanually-classified state of the seal.

After the classified state of the vacuum-packaged product, asrepresented at block 1226, is classified or obtained by theclassification decision-making process 1206, the classificationdecision-making process 1206 sends the classified stated of the seal tothe output device 1208. In the embodiments where the classificationdecision-making process 1206 is software operating on one or morecomputing devices, the output device 1208 can be a user interface outputdevice. In some embodiments, the outputting the classified state of theseal at block 1236 includes one or more of outputting the classifiedstate of the seal to a user via a user interface (e.g., a monitor, atouchscreen, etc.), communicating the classified state of the seal to anexternal device via a communications circuitry, or locally storing theclassified state of the seal in a database.

In any of the embodiments disclosed herein, the image data received forany one seal may include multiple forms of image data about the sameseal. For example, image data about a seal may include two images in thevisible light range of the same seal. In another embodiment, image dataabout a seal may include three images, one being a thermography image,one being an ultraviolet image and one be a photoelasticity image. Thesemultiple different forms of image data for the same seal may be passedthrough an artificial intelligent algorithm separately. If theartificial intelligent algorithm returns the same classified state forthe seal using multiple different forms of image data, then theconfidence level of the classification for that seal can be increasedsignificantly. In one example, if the artificial intelligent algorithmclassified one of the images as having a packaging article with animperfect seal at a 98% confidence level and classified the other imageas having a packaging article with an imperfect seal at a 96% confidencelevel, then the confidence level that the packaging article has animperfect seal may be greater than 99%. In another example, if theartificial intelligent algorithm classified one of the images as havinga non-defective seal at a 60% confidence level and classified the otherimage as having a non-defective seal at a 70% confidence level, then theconfidence level that the vacuum-packaged product is non-defective maybe 88%. Even though the confidence level using two images may besignificantly higher than either of the images alone, the combinedconfidence level from two images (e.g., 88%) may still be below apredetermined percentage of an acceptable degree of certainty (e.g.,95%), which may cause the packaging article to be flagged for manualclassification. In some further embodiments, multiple camera angles maybe used to image the seal on multiple surfaces and from multipleviewpoints, such as through the packaging article, at an angle from thepackaging article or directly above the packaging article. In someembodiments, two or more camera angles to image the same seal may beused. It will be apparent that the number of multiple forms of imagedata is not limited to two, but could be any number of forms of imagedata.

In some embodiments, not every area of increased or altered imageindicators or defects is necessary in detection and reporting, and auser or the computer apparatus may determine one or more thresholds tofacilitate identification of a defective seal. The threshold may bebased on any of the parameters detailed herein and may be predeterminedor applied as the result of an algorithm or model. A threshold value canbe set so that only altered image indicators and defects above thethreshold size are flagged for removal. For example, the threshold canbe set at a fluorescent region having a size of at least 2 millimetersin at least one direction (i.e., an altered fluorescence having a sizeof at least 2 millimeters in the machine direction and/or at least 2 mmin the transverse direction). Alternatively, the threshold can be set toa thermal image abnormality of at a size of at least 1 millimeter in atleast one direction, (i.e., an altered fluorescence of at least 1millimeter in at least one direction). In some embodiments, a thresholdvalue can be set at a predetermined surface area (e.g., an area of atleast 1 mm₂, 2 mm₂, 3 mm₂, or 4 mm₂). Such a threshold can be set evenif the system has the capability to see defects down to a size of as lowas 10 microns in at least one direction. The setting of the thresholdvalue is different from the capability of the machine vision system todetect a defect to at least a particular size in at least one direction.Rather, the setting of the threshold value is the setting of the minimumvalue of the size of the area which trigger the generation of the signalin response thereto. That threshold can be set at any desired value, andis different from the capability of the machine vision system to detectdefects down to at least a specified size.

In some embodiments, the algorithmic or model-based detecting meansdisclosed herein may be used to detect defects in the packaging article.For example, in some embodiment, the algorithmic or model-baseddetecting means described herein may detect an area of high fluorescencethat is higher than the majority of the seal, which may indicate a pleator other defect in the seal. In some embodiments, the algorithmic ormodel-based detecting means may detect a substantially even, constantfluorescence from the seal, indicating a strong seal. In someembodiments, areas of less or no fluorescence in the region of the sealmay indicate a discontinuity or weakness in the seal.

In some embodiments, the algorithmic or model-based detecting meansdisclosed herein may be used to detect defects in the packaging article.For example, in some embodiment, the algorithmic or model-baseddetecting means described herein may detect an area of high thermalradiation concentration that is higher than the majority of the seal,which may indicate a pleat or other defect in the seal. In someembodiments, the algorithmic or model-based detecting means may detect asubstantially even, constant thermal image from the seal, indicating astrong seal. In some embodiments, areas of less or no thermal radiationin the region of the seal may indicate a discontinuity or weakness inthe seal.

In some embodiments, the algorithmic or model-based detecting meansdisclosed herein may be used to detect defects in the packaging article.For example, in some embodiment, the algorithmic or model-baseddetecting means described herein may detect a visible defect underphotoelastic images.

Examples

In an example embodiment, the algorithm and image processing maycomprise some or all of the following steps: (1) capture an image; (2)apply de-noising to the image; (3) convert the image to greyscale; (4)apply thresholding to reduce the section of the image for consideration;(5) eliminate undesired areas of the image; (6) identify features in theremaining image corresponding to a faulty seal in the packaging article.Examples of each of these processing steps are shown and described inthe following figures.

Turning to FIG. 22, thermal image data of a sufficient seal is shown. Animpulse seal was formed in a packaging article. Impulse seals are knownto create non-linear or wavy seal areas. The image was captured 5seconds after seal creation. A uniform active cooling was utilized aftersealing and prior to image capture. Such a sufficient seal image can beused for the artificial intelligent algorithm as good seal images. FIG.23 is exemplary image of thermal image data of a faulty seal. An impulseseal was formed in a packaging article. The image was captured 5 secondsafter seal creation. A uniform active cooling was utilized after sealingand prior to image capture. The image, and similar faulty seal imagescan be used to train the artificial intelligent algorithm to demonstratedefects or insufficient seals.

FIG. 23 is an exemplary thermal image showing double straight seal witha localized seal pleat. An impulse seal was formed in a packagingarticle. The image was captured 5 seconds after seal creation. A uniformactive cooling was utilized after sealing and prior to image capture.The discontinuity in the thermal image along the seal area demonstrate acold and weak seal. Such images are useful in training the artificialintelligent algorithm as described herein.

FIG. 24 is an exemplary thermal image showing a double straight sealwith a weak seal area. An impulse seal was formed in a packagingarticle. The image was captured 5 seconds after seal creation. A uniformactive cooling was utilized after sealing and prior to image capture.The discontinuity in the thermal image along the seal area demonstrate acold and weak seal. Such images are useful in training the artificialintelligent algorithm as described herein.

Turning now to FIGS. 25 and 26 there is shown thermal image data imagesdemonstrating an acceptable seal. FIGS. 25 and 26 are taken of the sameseals at two distinct locations and two distinct points in time. Theseal area emits a fairly consistent thermal radiation along the sealwithout any hot or cold spots or areas. The heat in the seal area isresidual heat from the sealing process and dissipates over time. FIG. 25is an image taken 15 s after the heat sealing process. FIG. 26 is animage taken 20 s after the heat sealing. The time for capturing thethermal image after heat sealing can be adjusted depending on the film,packaging article, thickness of the film, heat seal temperature andmethod.

Referencing FIGS. 27 and 28 there is shown thermal image data imagesdemonstrating a defective seal. FIGS. 25 and 26 are taken of the sameseals at two distinct locations and two distinct points in time. Theseal area emits a fairly consistent thermal radiation along the sealwith the exception of the hot spots 3101, 3102, 3201 and 3202. The heatin the seal area is residual heat from the sealing process anddissipates over time. FIG. 27 is an image taken 15 s after the heatsealing process. FIG. 28 is an image taken 20 s after the heat sealing.The time for capturing the thermal image after heat sealing can beadjusted depending on the film, packaging article, thickness of thefilm, heat seal temperature and method.

Turning now to FIG. 29 is shown a seal area captured by aphotoelasticity imaging system, such as described herein. Thephotoelasticity image capture mechanism provides enhanced visioncapabilities as compared to standard image capture techniques. FIG. 29demonstrates an adequate seal without any defects.

FIGS. 30 and 31 demonstrate seal areas captured by a photoelasticityimaging system. The images improve the definition of the pleat defectsas compared to standard imaging. The vision inspection engine analyzesthe image for variations in adjacent pixels color gradients, sectioncomparison and the like. The vision inspection engine may compare anumber of pixels, or sections of images one or more of the following:mean, variable, skew, minimum, maximum, range or variations in the sealarea. Variations between pixels or section of the image can beindicative of a seal defect.

FIG. 32 demonstrates an ultraviolet image taken of a packaging articlehaving an optical brightener disposed therein. The optical brightenerfluoresces with more intensity at the seal areas containing pleatdefects. Additional material is bunched around the pleat defects causingadditional illumination intensity. The vision inspection engine analyzesthe image for variations in adjacent pixels color gradients, sectioncomparison and the like. The vision inspection engine may compare anumber of pixels, or sections of images one or more of the following:mean, variable, skew, minimum, maximum, range or variations in the sealarea. Variations between pixels or section of the image can beindicative of a seal defect.

CONCLUSION

Many modifications and other embodiments of the inventions set forthherein will come to mind to one skilled in the art to which theseinventions pertain having the benefit of the teachings presented in theforegoing descriptions and the associated drawings. Therefore, it is tobe understood that the inventions are not to be limited to the specificembodiments disclosed and that modifications and other embodiments areintended to be included within the scope of the appended claims.Moreover, although the foregoing descriptions and the associateddrawings describe example embodiments in the context of certain examplecombinations of elements and/or functions, it should be appreciated thatdifferent combinations of elements and/or functions may be provided byalternative embodiments without departing from the scope of the appendedclaims. In this regard, for example, different combinations of elementsand/or functions than those explicitly described above are alsocontemplated as may be set forth in some of the appended claims.Although specific terms are employed herein, they are used in a genericand descriptive sense only and not for purposes of limitation.

1. A process for monitoring seal quality of a packaging article,comprising: A) sealing a film to itself, another film, or a packagingsupport to form a packaging article by forming at least one seal area;B) acquiring image data of the at least one seal area with a visionsystem comprising an image capture device; C) assessing the image dataof the seal area with a vision inspection engine to verify thecontinuity of the seal, the strength of the seal, or both the continuityand strength of the seal area.
 2. The process according to claim 1,wherein the forming at least one seal area is formed by a heat generatedseal.
 3. (canceled)
 4. (canceled)
 5. The process according to claim 1,wherein the film comprises at least one layer containing afluorescence-based indicator.
 6. (canceled)
 7. The process according toclaim 5, wherein the vision system is an ultraviolet vision systemfurther comprising an ultraviolet light source.
 8. The process accordingto claim 5, wherein the ultraviolet vision system further comprises awhite light source.
 9. (canceled)
 10. (canceled)
 11. The processaccording to claim 5, further comprising the steps of: A) exposing thepackaging article to incident radiation to excite the fluorescence-basedindicator so that the fluorescence-based indicator fluoresces; B)acquiring image data of the fluorescence emitted from the seal area bythe packaging article, while the indicator is fluorescing.
 12. Theprocess of claim 11, wherein the fluorescence-based indicator comprisesat least one member selected from the group consisting ofultraviolet-indicator, infrared-indicator, dye, pigment, opticalbrightener, fluorescent whitening agent,2,2′-(2,5-thiophenylenediyl)bis(5-tert-butylbenzoxazole),hydroxyl-4-)p-tolylamino)anthracene-9,10-dione,2,5-thiophenediylbis(5-tert-butyl-1,3-benzoxazole), and anthraquinonedyestuff.
 13. (canceled)
 14. (canceled)
 15. The process according toclaim 11, wherein the vision inspection engine comprises a computingapparatus comprising computer executable instructions configured todetermine whether fluorescent electromagnetic energy emitted by theexcited fluorescence-based indicator is indicative of a defective seal.16. The process of claim 15 wherein the computer executable instructionscomprise at least one artificial intelligence algorithm selected fromthe group of supervised, unsupervised or semi-supervised methodology.17-19. (canceled)
 20. The process according to claim 11, whereindetermining that the seal is defective comprises determining at leastone of (i) that the film exhibits a higher or lower intensity offluorescence in a first region of the seal, relative to a second regionof the seal, (ii) that the film exhibits a higher or lower intensity offluorescence in a first region of the seal, relative to an expectedlevel fluorescence, or (iii) both (i) and (ii).
 21. The processaccording to claim 1 wherein the image data is time-delayed thermographydata captured at a time after the forming of at least one seal area andthe vision system is a thermography vision system comprising an infraredimaging device capable of capturing a temperature distribution based onthe amount of infrared radiation emitted from the seal area, wherein theimage data is taken between 2 and 30 seconds after the forming at leastone seal area. 22-29. (canceled)
 30. The process according to claim 1,wherein the vision system is a photoelasticity vision system comprising:(i) a first linear polarizer having a direction of polarization orientedin a first direction; (ii) a second linear polarizer have a direction ofpolarization oriented orthogonal to the first direction; (iii) a lightsource; and (iv) an imaging device. 31-36. (canceled)
 37. The processaccording to claim 1, wherein the vision system is a first vision systemand further comprising second vision system distinct from the firstvision system.
 38. The process according to claim 37, wherein the firstvision system is a thermography vision system comprising an infraredimaging device; and the second vision system is a photoelasticity visionsystem comprising: (i) a first linear polarizer having a direction ofpolarization oriented in a first direction; (ii) a second linearpolarizer have a direction of polarization oriented orthogonal to thefirst direction; (iii) a light source; and (iv) an image capture device.39. (canceled)
 40. The process according to claim 37, wherein the firstvision system is a thermography vision system comprising an infraredimaging device; and the second vision system is an ultraviolet visionsystem comprising an ultraviolet light source.
 41. (canceled)
 42. Theprocess according to claim 37, wherein the first vision system is anultraviolet vision system comprising an ultraviolet light source; andthe second vision system is a photoelasticity vision system comprising:(i) a first linear polarizer having a direction of polarization orientedin a first direction; (ii) a second linear polarizer have a direction ofpolarization oriented orthogonal to the first direction; (iii) a lightsource; and (iv) an image capture device. 43-59. (canceled)
 60. Theprocess according to claim 1, wherein the image data is selected fromthe group consisting of thermal image data, photoelasticity image dataand ultra violet fluorescence emitted image data.
 61. (canceled)
 62. Theprocess according to claim 1, wherein the image data of the at least oneseal area is captured by the image capture device at a speed of at least5 images per second. 63-66. (canceled)
 67. The process according toclaim 1, wherein the vision inspection engine assigns a seal score valueto the image data of the seal area. 68-70. (canceled)
 71. A system fordetecting a defective seal of a packaging article comprising: A) asealing mechanism configured to seal a film to itself, another film, ora packaging support to form a packaging article by forming at least oneseal area; B) a vision system comprising an image capture deviceconfigured to acquire image data of the at least one seal area with avision system; C) assessing the image data of the seal area with avision inspection engine to verify the continuity of the seal, thestrength of the seal, or both the continuity and strength of the sealarea. 72-93. (canceled)